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[Title of Document] Specification 
[Title of the Invention] 

INTERPOLATION PROCESSING APPARATUS AND RECORDING MEDIUM 
HAVING INTERPOLATION PROCESSING PROGRAM RECORDED THEREIN 
5 [Scope of Patent Claims] 
[Claim 1] 

An interpolation processing apparatus tlnat engages in 
processing on' image data which are provided in a 
colorimetric system constituted of first ^ nth (n ^ 3) 

10 color components and include color information 

corresponding to a single color component provided at each 
pixel to determine, an interpolation value equivalent to 
color information corresponding to the first color 
component for a pixel at which the first color component 

15 is missing, comprising: 

a means for interpolation value calculation that uses 
color information at nearby pixels located nearby an 
interpolation target pixel to undergo interpolation 
processing to calculate an interpolation value including, 

20 at least 

(1) a first term indicating local average information 
with regard to the first color component 

(2) a second term indicating curvature information with 
regard to a color component matching a color component at 

25 the interpolation target pixel and 
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(3) a third term indicating curvature information with 
regard to a color component other than the color component 
at the interpolation target pixel. 

[Claim 2] 

5 ' An interpolation processing apparatus according to 

claim 1, further comprising: 

a first means for similarity judgment that judges 
degrees of similarity to the interpolation target pixel 
along at least two directions in which nearby pixels with 
10 color information corresponding to the first color 

component are connected with the interpolation target 
pixel; and 

a second means for similarity judgment that judges 
degrees of similarity to the interpolation target pixel 
15 along at least two directions other than the directions in 
which the degrees of similarity are judged by said first 
means for similarity judgment, wherein: 

said means for interpolation value calculation 
selects a direction along which nearby pixels having color 
20 information to be used to calculate said first term and 

said second term are set based upon results of a judgment 
made by said first means for similarity judgment; and 

said means for interpolation value calculation 
selects a direction along which nearby pixels having color 
25 information to be used to calculate said third term are 
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set based upon results of a judgment made by said second 
means for similarity judgment. 
[Claim 3] 

An interpolation processing apparatus that engages in 
processing on image data which are provided in a 
colorimetric system constituted of first nth (n ^ 3) 
color components and include color information 
corresponding to a single color component provided at each 
pixel to determine an interpolation value equivalent to 
color information corresponding to the first color 
component for a pixel at which the first color component 
is missing, comprising: 

a means for interpolation value calculation that 
calculates an interpolation value including at least two 
terms, i.e., a first term and a second term by using color 
information at nearby pixels set nearby an interpolation 
target pixel to undergo interpolation processing; 

a first means for similarity judgment that' judges 
degrees of similarity to the interpolation target pixel 
along at least two directions in which nearby pixels 
having color information corresponding to the first color 
component are connected to the interpolation target pixel; 
and 

a second means for similarity judgment that judges 
degrees of similarity to the interpolation target pixel 
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along at least two directions other than the directions in 
which the degrees of similarity are judged by said first 
means for similarity judgment, wherein: 

said means for interpolation value calculation 
5 selects a direction along which pixels having color 

information to be used to calculate said first term are 
set based upon results of a judgment made by- said first 
means for similarity judgment and selects a direction 
along which pixels having color information to be used to 
10 calculate said second term are set based upon results of a 

judgment made by said second means for similarity judgment. 
[Claim 4] 

An interpolation processing apparatus according to 
claim 3, wherein 

15 said means for interpolation value calculation 

calculates a first term containing local average 
information with regard to the first color component and 
curvature information with* regard to a color component 
matching a color component at the interpolation target 

20 pixel, and a second term containing curvature information 
with regard to a color component other than the color 
component at the interpolation target pixel to calculate 
the interpolation value. 
[Claim 5] 

25 An interpolation processing apparatus according to 
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claim 2, wherein: 

.when image data are provided in a colorimetric system 
constituted of first third color components with the 
first color component achieving a higher spatial frequency 
5 than the second color component and the third color 

component, the first color component set in a checker- 
board pattern, the second color component and the third 
color component each set in a line sequence between pixels 
at which color information corresponding to the first 

10- color component is present and information corresponding 

to the second color component present at the interpolation 
target pixels- 
said first means for similarity judgment calculates 
similarity degrees manifested by the interpolation target 

15 pixel along two directions, i . e a- vertical direction and 
a horizontal direction, in which nearby pixels with color 
information corresponding to the first color component are 
connected to the interpolation target pixel and makes a 
judgment with regard to degrees of similarity manifested 

20 by the interpolation target pixel along the vertical 
direction and the horizontal direction based upon a 
difference between said similarity degrees; 

said second means for similarity judgment calculates 
similarity degrees manifested by the interpolation target 

25 pixel along two diagonal directions in which nearby pixels 
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with color information corresponding to the third color 
component are connected to the interpolation target pixel 
and makes a. judgment with" regard to degrees of similarity 
manifested by the interpolation target pixel along the two 
diagonal directions based upon a difference between said 
similarity degrees; and 

said means for interpolation value calculation 
selects color information corresponding to the first color 
component or color information corresponding to the third 
color component to be used in conformance to the degrees 
of similarity j^udged by said second means for similarity 
judgment when calculating said third term. 
[Claim 6] . 

An interpolation processing apparatus according to 
claim 5, wherein: 

said means for interpolation value calculation 
calculates the third term based upon color information 
corresponding to the first color component if said second 
means for similarity judgment judges that roughly equal 
degrees of similarity manifest along the two diagonal 
directions and calculates the third term based upon color 
information corresponding to the third color component if 
said second means for similarity judgment judges that a 
higher degree of similarity manifests along one of the two 
diagonal directions compared to the other diagonal 
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[Claim 7]' 

An interpolation processing apparatus according to 
claim 3, wherein: 
5 when image data are provided in a colorimetric system 

constituted of first third color components with the 
first color component achieving a higher spatial frequency 
than the second color component and the third color 
component;, the first color component set in a checker- 

10 board pattern, the second color component and the third 

color component each set in a line sequence between pixels 
at which color information corresponding to the first 
color component is present and information corresponding 
to the second color component present at the interpolation 

15 target pixels- 
said first means for similarity judgment calculates 
similarity degrees manifested by the interpolation target 
pixel along two directions, i.e., "a vertical direction and 
a horizontal direction, in which nearby pixels with color 

20 information corresponding to the first color component are 
connected to the interpolation . target pixel and makes a 
judgment with regard to degrees of similarity manifested 
by the interpolation target pixel along the vertical 
direction and the horizontal direction based upon a 

25 difference between said similarity degrees; 



9 



Translation of 
JP11~363007 



said second means for similarity judgment calculates 
similarity degrees manifested by the interpolation target 
pixel along two diagonal directions in which nearby pixels 
with color information corresponding to the third color 
5 component are connected to the interpolation target pixel 
and makes a judgment with regard to degrees of similarity 
manifested by the interpolation target pixel along the two 
diagonal directions based upon a difference between said 
similarity degrees; and 

10 said means for interpolation value calculation 

selects color information corresponding to the first color 
component or color information corresponding to the third 
color component to be used in conformance to the degrees 
of similarity judged by said second means for similarity 

15 judgment when calculating said second term. 
[Claim 8] 

An interpolation processing apparatus according to 
claim 7, wherein: 

said means for interpolation value calculation 
20 calculates the second term based upon color information 

corresponding to the first color component if said second 
means for similarity judgment judges that roughly equal 
degrees of similarity manifest along the two diagonal 
directions and calculates the second based upon color 
25 information corresponding to the third color component if 
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said second means for similarity judgment judges that a 
higher degree of similarity manifests along one of the two 
diagonal directions compared to the other diagonal 
direction. 
5 [Claim 9] 

An interpolation processing apparatus according to 
claim 5 . or 7, wherein: 

said first means for similarity judgment judges that 
roughly equal degrees of similarity manifest along the 
10 vertical direction and the horizontal direction if a 
difference between the similarity degrees along the, 
vertical direction and the- horizontal direction is smaller 
than a specific threshold value; and 

said second means for similarity judgment judges that 
15 roughly equal degrees of similarity manifest along the two 
diagonal directions if a difference between the similarity 
degrees along the two diagonal directions is smaller than 
a specific threshold value.' 
[Claim 10] 

20 An interpolation processing apparatus according to 

claim 5 or 7, wherein: 

said first means for similarity ^judgment calculates 
the similarity degrees "along the vertical direction and 
the horizontal direction by using color information 

25 corresponding to a plurality of color components for a 
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single interpolation target pixel; and 

said second means for similarity- judgment calculates 
the similarity degrees along the two diagonal directions 
by using color information corresponding to a plurality of 
5 color components for a single interpolation target pixel. 
[Claim 11] 

An interpolation processing apparatus according to 
claim 10;- wherein: 

said second means for similarity judgment calculates 
10 a similarity degree manifesting along each of the two 
diagonal directions through weighted addition of: 
(1) a similarity degree component constituted of color 
information corresponding to the first color component 
alone; 

15 (2) a similarity degree component constituted of color 
information corresponding to the second color component 
alone; 

(3) a similarity degree component constituted of color 
information corresponding to the third color component 

20 alone; and 

(4) a similarity degree component constituted of color 
information corresponding to the second color comp9nent 
and the third color component. 

[Claim 12] 

25 An interpolation processing apparatus according to 
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claim 5 or 7, wherein: 

said first means for' similarity judgment calculates 
similarity degrees along the vertical direction and the 
horizontal direction for each pixel and makes a judgment 
5 on similarity manifested by the interpolation target pixel 
along the vertical direction and the horizontal direction 
■ based upon differences in similarity degrees manifesting 
at nearby pixels as well as at the interpolation target 
pixel; and 

10 - said second means for similarity judgment calculates 

similarity degrees along the two diagonal directions for 
each pixel and makes a judgment on similarity manifested 
by the interpolation target pixel along the two diagonal 
directions based upon differences in similarity degrees 

15 manifesting at nearby pixels as well as at the 
interpolation target pixel. 
[Claim 13] 

An interpolation processing apparatus that engages 'in 
processing on image data which are provided in a 

20 colorimetric system constituted of first nth (n ^ 3) 
color components and include color information 
corresponding to a single color component provided at each 
pixel to determine an interpolation value equivalent to 
color information corresponding to the first color 

25 component for a pixel at which the first color component 
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. is missing, comprising: 

a means for first term calculation that calculates a 
first term representing average information of the first 
color component with regard to an interpolation target 
5 pixel to undergo interpolation processing by using color 
information corresponding to color components at nearby 
pixels set nearby the interpolation target pixel; 

a means for second term calculation that calculates a 
second term representing local curvature information with 

10 regard to a color component matching the color component 
at the interpolation target pixel by using color 
information corresponding to color components at pixels 
each containing a color component same as the 
interpolation target pixel; and 

15 a means for interpolation value calculation that 

calculates an interpolation value by adding said second ' 
term multiplied by a weighting coefficient constituted of 
color information corresponding to 'a plurality of color 
components at nearby pixels and the interpolation target 

20 pixel to said first term. 
[Claim 14] 

An interpolation processing apparatus according to 
claim 13, wherein: 

said means for interpolation value calculation uses 
25 color information corresponding to a plurality of color 
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components provided at the interpolation target pixel and 
at a plurality of pixels set along a predetermined 
direction relative to the interpolation target pixel' to 
ascertain inclinations manifesting in color information 
corresponding to the individual color components along the 
direction and calculates said weighting coefficient in 
conformance to a correlation manifesting among the 
inclinations in the color information corresponding to the 
individual color components. 
[Claim 15] 

An interpolation processing apparatus that implements 
processing for supplementing a color component value at a 
pixel at which information corresponding to a color 
component is missing in image data provided in a 
colorimetric system constituted of a luminance component 
and the color component, with the luminance component 
having a higher spatial frequency than the color component 
and the luminance component present both at pixel's having 
information corresponding to the color component and at 
pixels lacking information corresponding to the color 
component, comprising: 

a means for hue value calculation that calculates hue 
values at a plurality of pixels located near an 
interpolation target pixel to undergo interpolation 
processing and having both the luminance component and the 
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color component by using luminance component values and 
color component values at the individual pixels; 

a means for hue value interpolation that calculates a 
hue value at the interpolation target pixel by using a 
median of the hue values at the plurality of pixels 
calculated by said means for hue value calculation; and 

a means for color conversion that interpolates a 
color component at the interpolation target pixel by using 
the luminance component at the interpolation target pixel 
to convert the hue value at the interpolation target pixel 
calculated by said means for hue value interpolation to a 
color component . 
[Claim 16] 

An interpolation processing apparatus that implements 
processing for supplementing a luminance component at a 
pixel at which information corresponding to a luminance 
component is missing and supplementing a color component 
at a pixel at which information corresponding ' to a color 
component is missing, on image data provided in a 
colorimetric system constituted of the luminance component 
and the color component, with the luminance component 
having a higher spatial frequency than the color component 
and a given pixel having only information corresponding to 
either the luminance component or the color component, 
comprising: 
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a means for luminance component interpolation that 
interpolates a luminance component at a luminance 
component interpolation target pixel to undergo luminance 
component interpolation processing by using at least 
either "similarity manifesting between the luminance 
component interpolation target pixel and a pixel near the 
luminance component interpolation target pixel" or "a 
plurality of color components within a local area 
containing the luminance component interpolation target 
pixel"; 

a means for hue value calculation that calculates hue 
values at a plurality of pixels located near an 
interpolation target pixel to undergo color component 
interpolation processing, having color component values 
and having luminance component values interpolated by said 
means for luminance component interpolation, by using the 
luminance component values and color component values at 
the individual pixels; 

a means for hue value interpolation that calculates a 
hue value for the interpolation target pixel by using a 
median of the hue values at the plurality of pixels 
calculated by said means for hue value calculation; and 

a means for color conversion that interpolates a 
color component value for the interpolation target pixel 
by using the luminance component value at the 
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interpolation target pixel to convert the hue value at the 
interpolation target pixel calculated by said means for 
hue value interpolation . to a color component value. 
[Claim 17] 

5 An interpolation processing apparatus according to 

claim 15 or 16, wherein: 

when the luminance component in the image data 
corresponds to a green color component and the color 
component in the image data corresponds to a red color 

10 component and a blue color component, 

said means for hue value interpolation calculates a 
hue value for the interpolation target pixel by using a 
median of hue values containing the red color component at 
pixels near the interpolation target pixel if the green 

15 color component is present but the red color component is 
missing at the interpolation target pixel and calculates a 
hue value for the interpolation target pixel by using a 
median of hue values containing the blue color component' 
at pixels near the interpolation target pixel if the green 

20 color component is present but the blue color component is 
missing at the interpolation target pixel. 
[Claim 18] 

An interpolation processing apparatus according to 
claim 15 or 16, wherein: 
25 when the luminance component in the image data 
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corresponds to a green color component and the color 
component in the image data corresponds to a red color 
component and a blue color component, 

said means for hue value interpolation calculates a 
5 hue value for the interpolation target pixel by using a 

median of hue values containing the red color component at 
pixels set near the interpolation target pixel if the blue 
color component is present but the red color component is 
missing at the interpolation target pixel. 
10 [Claim 19] 

An interpolation processing apparatus according to 
claim 15 or 16, wherein: 

when the luminance component in the image data 
corresponds to a green color, component and the color 
15 component in the image data corresponds to a red color 
component and a blue color component, 

said means for hue value interpolation calculates a 
hue value for the interpolation target pixel by using' a 
median of hue values containing the blue color component 
20 at pixels set near the interpolation target pixel if the 
red color component is present but the blue color 
component is missing at the interpolation target pixel. 
[Claim 20] 

An interpolation processing apparatus according to 
25 any one of claims 15 through 19, with a color component 
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missing at the interpolation target pixel present at only 
one pixel among four pixels set symmetrically along the 
vertical direction and the horizontal direction, wherein 
said means for hue value interpolation comprises: 

a first hue value interpolation unit that calculates, 
a hue value for the interpolation target pixel by using a 
median of hue values at a plurality of diagonally adjacent 
pixels if the hue values of the plurality of diagonally 
adjacent pixels adjacent to the interpolation target pixel 
along diagonal directions have been calculated by said 
means for hue value calculation; and 

a second hue value interpolation unit that calculates 
a hue value for the interpolation target pixel by using a 
median of hue values at a plurality of vertically and 
horizontally adjacent pixels if the hue values of the 
plurality of vertically and horizontally adjacent pixels 
adjacent to the interpolation target pixel in the vertical 
direction and the horizontal direction have been 
calculated by said means for hue value calculation or said 
first hue value interpolation unit. 
[Claim 21] 

A recording medium having recorded therein an 
interpolation processing program to implement on a 
computer processing for determining an interpolation value 
equivalent to color information corresponding to a first 
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color component missing at a pixel, on image data provided 
in a colorimetric system constituted of first ^ nth (n ^ 
3) color components with color information corresponding 
to a single color component present at each pixel, said 
interpolation processing program comprising: 

an interpolation value calculation step in which an 
interpolation value including, at least 

(1) a first term indicating local average information 
with regard to the first color component 

(2) a second term indicating curvature information with 
regard to a color component matching a color component at 
the interpolation target pixel and 

(3) a third term indicating curvature information with 
regard to a color component other than the color component 
at the interpolation target pixel. 

[Claim 22] 

A recording medium having recorded therein an 
interpolation processing program to implement on a 
computer processing for determining an interpolation value 
equivalent to color information corresponding to a first 
color component missing at a pixel, on image data provided 
in a colorimetric system constituted of first - nth (n ^ 
3) color components with' color information corresponding 
to a single color component present at each pixel, said 
interpolation processing program comprising: 



Translation of 
JPll-363007 



an interpolation value calculation step in which an 
interpolation value including at least two terms, i.e., a 
first term and a second term is calculated by using color 
information at nearby pixels set nearby an interpolation 
5 target pixel to undergo interpolation processing; 

a first similarity judgment step in which degrees of 
similarity to the interpolation target pixel are judged 
along at least two directions in which nearby pixels 
having color information corresponding to the first color 
10 component are connected with the interpolation target 
pixel; and 

a second similarity judgment step in which degrees of 
similarity to the interpolation target pixel are judged 
along at least two directions other than the directions 

15 along which the degrees of similarity are judged in said 
first similarity judgment step, wherein: 

in said interpolation value calculation step, a 
direction in which pixels * having color information to be 
used to calculate said first term are set is selected 

20 based upon results of a judgment made in said first • 

similarity judgment step and a direction' in which pixels 
having color information to be used to calculate said 
second term are set is selected based upon results of a 
judgment made in said second similarity judgment step. 

25 [Claim 23] 
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A recording medium having recorded therein an 
interpolation processing program to implement on a 
computer processing for determining an interpolation value 
equivalent to color information corresponding to a first 
color component missing at a pixel, on image data provided 
in a colorimetric system constituted of first nth (n ^ 
3) color components with color information corresponding 
to a single color component present at each pixel, said 
interpolation processing program comprising: 

a first term calculation step in which a first term 
representing average information of the first color 
component with regard to an interpolation target pixel to 
undergo interpolation processing is calculated by using 
color information corresponding to a color component at 
nearby pixels set nearby the interpolation target pixel; 

a second term calculation step in which a second term 
representing local curvature information with regard to a 
color component matching, the color component at the 
.interpolatipn target pixel is calculated by using color 
information corresponding to a color component at pixels 
each containing a color component same as the 
interpolation target pixel; and 

an interpolation value calculation step in which an 
interpolation value is calculated by adding said second 
term multiplied by a weighting coefficient constituted of 
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color information corresponding to a plurality of color 
components provided at nearby pixels and the interpolation 
target pixel to the first term. 
[Claim 24] 

5 A recording medium having recorded therein an 

interpolation processing program for implementing on a 
computer processing supplementing a color component value 
at a pixel at which information corresponding to a color 
component is missing, on image data provided in a 

10 colorimetric system constituted of a luminance component 
and the color component, with the luminance component 
having a higher spatial frequency than the color component 
and the luminance component present both at pixels having 
information corresponding to the color component and at 

15 pixels lacking information corresponding to the color 
component; said interpolation processing program 
comprising : 

a hue value calculation step in' which hue values 'for 
a plurality of pixels near an interpolation target pixel 

20 to undergo interpolation processing and having information 
corresponding to both the luminance component and the 
color component are calculated by using luminance 
component values and color component values at the 
individual pixels; 

25 a hue value interpolation step in which a hue value 
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for the interpolation target pixel is calculated by using 
a median of the hue values at the plurality of pixels 
calculated in the hue value calculation step; and 

a color conversion step in which a color component 
value at the interpolation target pixel is interpolated by 
using a value indicated by the luminance component present 
at the interpolation target pixel ' to convert the hue value 
of the interpolation target pixel calculated in the hue 
value interpolation step to a color component value. 
[Claim 25] 

A recording medium having recorded therein an 
interpolation processing program for implementing on a 
computer processing for supplementing a luminance 
component value at a pixel at which information 
corresponding to a luminance component is missing and a 
color component value at a pixel at which information 
corresponding to a color component missing, on image data 
provided in a colorimetric system' constituted of the 
luminance' component and the color component, with the 
luminance component having a higher spatial frequency than 
the color component and information corresponding to 
either the luminance component or the color component 
present at each pixel, said interpolation processing 
program comprising: 

a luminance component interpolation step in which a 
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luminance component value is interpolated for a luminance 
component interpolation target pixel to undergo luminance 
component interpolation processing by using at least 
either "similarity between the luminance component 
5 interpolation target pixel and a pixel near the luminance 
component interpolation target pixel" or "information 
corresponding to a plurality of color components within a 
local area containing the luminance component" 
interpolation target pixel"; 

10 a hue value calculation step in which hue values at a 

plurality of pixels located near an interpolation target 
pixel to undergo color component interpolation processing, 
having color component values and having luminance 
component values interpolated in said luminance component 

15 interpolation step are calculated by using the luminance 
component values and color component values at the 
individual pixels; 

a hue value interpolation step in which a hue value 
for the interpolation target pixel is calculated by using 

20 a median of the hue values at the plurality of pixels 
calculated in the hue value calculation step; and 

a color conversion step in which a color component 
value is interpolated for the interpolation target pixel 
by using the luminance ■ component value at the 

25 interpolation target pixel to convert the hue value at the 
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interpolation target pixel calculated in said hue value 
interpolation step to a color component value. 
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[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] 

The present invention relates to an interpolation 
processing apparatus that engages in interpolation 
processing on color image data to supplement a color 
component and a luminance component missing in pixels and 
a recording medium having an interpolation processing 
program for achieving the interpolation processing on a 
computer that can be read by a computer. 

[0002] 

[Prior Art] 

Some electronic cameras generate color image data by 
employing an image-capturing sensor having three color (R, 
G and B: red, green and blue) color filters provided at 
specific positions (e.g., a Bayer array). In such an 
electronic camera in which the individual pixels at the 
image-capturing sensor each output color " information 
corresponding to only a single color component, it is 
necessary to implement interpolation processing to obtain 
color information corresponding to all the color 
components for each pixel. 
[0003] 

In an interpolation processing method proposed in the 
prior art, spatial similarity manifested by an 
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interpolation target pixel undergoing the interpolation 
processing is judged and an interpolation value is 
calculated by using the color information output from 
pixels positioned along the direction in which a high 
5 degree of, similarity is manifested. 

For instance, in the art disclosed in USP No. 
5,629,734, a green color interpolation value G5 for the 
interpolation target pixel is calculated through one 
formula among formula 1 through formula 3 when the color 

10 information corresponding to individual pixels is provided 
as shown below, with A5 representing the color information 
at the interpolation target pixel (a pixel with the green 
color component missing), Al, A3, ' A7 and A9 representing 
color information from pixels provided with color filters 

15 in the same color as the color of -the filter at the ^ 
interpolation target pixel and G2, G4, G6 and G8 
representing color information from pixels provided with 
green color filters. 
[0004] 

20 Al 
G2 

A3 G4 A5 G6 A7 
G8 
A9 

25 If a marked similarity manifests along the horizontal 



29 



Translation of 
JPll-363007 



direction, the green color interpolation value G5 for the 
interpolation target pixel is calculated through; 
G5= (G4+G6) /2+ (-A3 + 2A5-A7) /4 ... (formula 1). 
If a marked similarity manifests along the vertical 
5 direction, , the green color interpolation value G5 for the 
interpolation target pixel is calculated through; 
G5= (G2+G8) /2+ (-A1 + 2A5-A9) /4 ... (formula 2). 

If roughly equal degrees of similarity manifest along the 
horizontal direction and the vertical direction, the green 
10 color interpolation value G5 for the interpolation target 
pixel is calculated through 

G5= (G2+G4+G6+G8) /4+ (-A1-A3 + 4A5-A7-A9) /8 ... (formula 3), 
It is to be noted that in order to simplify the subsequent 
explanation, the first terms ( (G4 + G6)/2, (G2 + G8)/2) in 
15 formulae 1 and 2 are each referred to as a primary term 
and that second terms ((-A3 + 2A5 -A7)/4 , (-A1 + 2A5 - 
A9)/4) in formulae 1 and 2 are each referred to as a 
correctional ■ term. 
[0005] 

20 In USP No. 5,629,734, assuming that the image data 

undergoing the interpolation processing manifest marked 
similarity along the horizontal direction with A3, G4, A5, 
G6 and A7 provided as indicated with # marks in FIG. 15, 
A4 representing the average of A3 and A5, and A6 

25 representing the average of A5 and A7, the value of the 
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correctional term in formula 1 is equivalent to the vector 
quantity (a in FIG. 15) representing the difference 
between A5 and the average of A4 and A6. In addition, the 
green color interpolation value G5 is equivalent to a 
5 value achieved by correcting the average of the values 
indicated by the color information from pixels adjacent 
along the horizontal direction (corresponds to the value 
of the primary term in formula 1) by a. 
[0006] 

10 In other words, in the art disclosed in USP No. 

5,629,734, a green color interpolation value is calculated 
by assuming that the color difference between the green 
color component and the color component (the red color 
component or the blue color component) at the 

15 interpolation target pixel is constant ((A4-G4), (A5-G5) 
and (A6-G6) in FIG. 15 match) and correcting the average - 
of the values indicated by the color information from the 
pixels that are adjacent along 'the direction in which a 
high degree of similarity is manifested with color 

20 information corresponding to the same color component as 
that of the interpolation target pixel. 
[0007] 

[Problems to be solved by the Invention] 

Optical systems such as lenses are known to manifest 
25 magnification chromatic aberration. For instance, if 
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there is magnification chromatic aberration at the 
photographic lens of an electronic camera having an image- 
capturing sensor provided with color filters in three 
colors, i.e.,. R, G and B, arranged in a Bayer array, 
5 images corresponding to the red color component and the 
blue color component are formed at positions slightly 
offset from the position at which the image corresponding 
to the green color component is formed, as shown in FIG. 
16. 
10 [0008] 

If the photographic lens is free of any magnification 
chromatic aberration and color information corresponding 
to the individual pixels is provided as indicated by the # 
marks in FIG. 17 (1) (the image data undergoing the 

15 interpolation processing manifest marked similarity along 
the horizontal direction, the color information 
corresponding to the green color component indicates a 
constant value and the values indicated by the color 
information corresponding to the red color component and 

20 the color information corresponding to the blue color 
component both change gently in the vicinity of the 
interpolation target pixel (the pixel at which A5 is 
present), the value of the correctional term in formula 1 
is 0 and, as a result, the average of G4 and G6 (the 

25 primary term) is directly used as the green color 
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interpolation value G5 without correction. 
[0009] - 

However, when A3, A5 and A7 each represent color 
information corresponding to the red color component and 
5 each set of color information corresponding to the red 

color component is offset by one pixel to the right due to 
a magnification chromatic , aberration at the photographic 
lens, the color information from the individual pixels 
undergoes "a change as shown in FIG. 17 (2) . Consequently, 

10 the value of the correctional term- in formula 1 is not 0 

and the primary term is over-corrected (hereafter referred 
to as an "over-correction") in such a case, resulting in 
the green color interpolation value G5 that should be 
similar to the. values indicated by G4 and G6 becoming 

15 larger than the G4 and G6 values (hereafter -this 

phenomenon is referred to as an "overshoot"). If, on the 
other hand, A3, A5 and A7 each represent color information 
corresponding to the blue color component ' and each set of 
-color information corresponding to the blue color 

20 component is offset by one pixel to the left due to a 

magnification chromatic aberration, the color information 
from the individual pixels undergoes a change as shown in 
FIG. 17 (3) . Thus, the value of the correctional term in 
formula 1 is not 0, resulting in the green color 

25 interpolation value' G5 that should be similar to the G4 
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and G6 values becoming smaller than those corresponding to 
G4 and G6 (hereafter this phenomenon is referred to as an 
"undershoot") through an over-correction. 
[0010] 

5 In other words, the art disclosed in USP No - 

5,629,734 poses a problem in that color artifacts occur in 
the color image obtained through the interpolation 
processing due to a magnification chromatic aberration. 
An over correction also occurs at a color boundary 

10 where the color difference changes as well as when there 
is a magnification chromatic aberration. . For instance, 
the color information corresponding to the individual 
pixels is provided as indicated by the #' marks in FIG. 18 
(1) and (2) (when the color information corresponding to 

15 the green color component is constant and the values 

identified by the color information corresponding to the . 
red color component or the blue color component change 
drastically near the interpolation target pixel (the pixel 
at which A5 is present)), the value of the correctional 

20 term in formula 1 is not 0 and, an overshoot or an 

undershoot occurs due to an over correction with regard to 
the green color interpolation value G5, which should be 
similar to the values indicated by G4 and G6. 
[0011] 

25 Thus, in a color boundary where the color difference 
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changes, a color artifact occurs as a result of 
interpolation processing even if there is no magnification 
chromatic aberration. It is to be noted that such a color 
artifact as that described above may occur when 
calculating a red color interpolation value or a blue 
color interpolation value as well as when calculating a 
green color interpolation value. 
[0012] 

An object of the present invention according to 
claims i through 18 is to provide an interpolation 
processing apparatus capable of preventing occurrence of 
color artifacts and an object of the present invention 
according to claims 19 through 23 is to provide a 
recording medium having recorded therein ah interpolation 
processing program with which occurrence of color 
artifacts can be prevented. 

More specifically, an object of the present invention 
according to claims 1 through 14 arid 19 through 21 is to 
suppress the occurrence of color artifacts by reducing the 
problems of the prior art while retaining the advantages 
of the interpolation processing in the prior art and 
reducing the degree of the adverse effect of magnification 
chromatic aberration . 
[0013] 

[Means for solving the Problems] 
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An interpolation processing apparatus according to 
claim 1 engages in processing on image data which are 
provided in a colorimetric system constituted of first 
nth (n ^ 3) color components and include color information 
5 corresponding to a single color. component provided at each 
pixel to determine an interpolation value equivalent to 
color information corresponding to the first color 
component for a pixel at which the first color component 
is missing, and comprises a means for interpolation value 
10 calculation that uses color information at nearby pixels 
located nearby an interpolation target pixel to undergo 
interpolation processing to calculate an interpolation 
value including, at least 

(1) a first term indicating local average information 
15 with regard to the first color component 

(2) a second term indicating curvature information with 
regard to a color component matching a color component at 
the interpolation target pixel and 

(3) a third term indicating curvature information with 

20 regard to a color component other than the color component 
at the interpolation target pixel. 
[0014] 

Namely, the invention according to claim 1 calculates 
an interpolation value by correcting the "first term 
25 indicating local average information with regard to the 
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first color component" with the "second term indicating 
curvature information with regard to a color component 
matching a color component at the interpolation target 
pixel" and the "third term indicating curvature 
5 information with regard to a color component other than 
the color component at the interpolation target pixel". 

An interpolation processing apparatus according to 
claim 2 is achieved by that, in the interpolation 
processing apparatus according to claim 1, there are 

10 further provided a first means for similarity judgment 
that judges degrees of similarity to the interpolation 
target pixel along at least two directions in which nearby 
pixels with color information corresponding to the first 
color component are connected with the interpolation 

15 target pixel, and a second means for similarity judgment 
that judges degrees of similarity to the interpolation 
target pixel along at least two directions other than the 
directions in which the degrees of simi-larity ' are judged 
by said first means for similarity judgment; said means 

20 for interpolation value calculation selects a direction 
along which nearby pixels having color information to be 
used to calculate said first term and said second term are 
set based upon results of a judgment made by said first 
means for similarity judgment; and said means for 

25 interpolation value calculation selects a direction along 
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which nearby pixels having color - information to be used to 
calculate said third term are set based upon results of a 
judgment made by said second means for similarity judgment. 
[0015] 

Namely, in .the invention according to claim 2, since 
directions along which the second means for similarity 
judgment judges degrees of similarity are different from 
directions along which the first means for similarity 
judgment judges degrees of similarity, a direction along 
which color information to be used to calculate the first 
term and the second term exists does not agree with a 
direction along which color information to be used to 
calculate the third term exists. 

Therefore, according to the invention of claim 2, the 
interpolation value can be calculated by using color 
information at nearby pixels located along a plurality of 
directions. Also, according to the invention of claim 2, 
the first term through the third term can be calculated by 
using color information at nearby pixels set along the 
direction in which a high degree of similarity is 
manifested or through weighted synthesis of color 
information from nearby pixels located along a plurality 
of directions, which is performed in correspondence to 
varying degrees of similarity. 
[0016] 
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An interpolation processing apparatus according to 
claim 3 engages in processing on image data which are 
provided in a colorimetric system constituted of first 
nth (n ^ 3) color components and include color information 
5 corresponding to a single color component provided at each 
pixel to determine an interpolation value equivalent to 
color information corresponding to the first color 
component for a pixel at which the first color component 
is missing, and comprises: a means for interpolation value 

10 calculation that calculates an interpolation value 

including at least two terms, i.e., a first term and a 
second term by using color information at nearby pixels 
set nearby an interpolation target pixel to undergo 
interpolation processing; a first means for similarity 

15 judgment that judges degrees of similarity to the 

interpolation target pixel along at least two directions 
in which nearby pixels having color information 
corresponding to the first color component are connected' 
to the interpolation target pixel; and a second means for 

20 similarity judgment that judges degrees of similarity to 
the. interpolation target pixel along at least two 
directions other than the directions in which the degrees 
of similarity are judged by said first means for 
similarity judgment, and said means for interpolation 

25 value' calculation selects a direction along which pixels 
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having color information to be used to calculate said 
first term are set based upon results of a judgment made 
by said first means for similarity judgment and selects a 
direction along which pixels having color information to 
be used to calculate said second term are set based upon 
results of a judgment made by said second means for 
similarity j udgment . 
[0017] 

Namely, in the invention according to claim 3, since 
directions along which the second means for similarity 
judgment judges degrees of similarity are different from 
directions along which the first means for similarity 
judgment judges degrees of similarity, a direction along 
which color information to be used to calculate the first 
term exists does not agree with a direction along which 
color information to be used to calculate the second term 
exists . 

Therefore, according to the invention of claim 3, the 
interpolation value can be calculated by using color 
information at nearby pixels located along a plurality of 
directions. Also, according to the invention of claim 3, 
the first term and the second term can be calculated by 
using color information at nearby pixels set along the 
direction in which a high degree of similarity is 
manifested or through weighted synthesis of color 
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information from nearby pixels located along a plurality 
of directions, which is performed in correspondence to 
varying degrees of similarity. 
[0018]. 

An interpolation processing apparatus according to 
claim 4 is achieved by that in the interpolation 
processing apparatus according to claim 3, said means for 
interpolation value calculation calculates a first term 
containing local average information with regard to the 
first color component and curvature information with 
regard to a color component matching a color component at 
the interpolation target pixel, and a second term 
containing curvature information with regard to a color - 
component other than the color component at the 
interpolation target pixel to calculate the interpolation 
value . 
[0019] 

Namely, the invention according to claim 4 calculates 
the interpolation value by correcting the "local average 
information with regard to the first color component" with 
the "curvature information with regard to a color 
component matching a color component at the interpolation 
target pixel" and the "curvature information with regard 
to a color component other than the color component at the 
interpolation target pixel". 
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An interpolation processing apparatus according to 
claim 5 is achieved by that in the interpolation 
processing apparatus according to claim 2: when image data 
are provided in a colorimetric system constituted of first 
5 third color components with the first color component 

achieving a higher spatial frequency than the second color 
component and the third color component, the first color 
component set in a checker-board pattern, the second color 
component and the third color component each set in a line 

10 sequence between pixels at which color information 

corresponding to the first color component is present and 
information corresponding to the second color component 
present at the interpolation target pixel; said first 
means for similarity judgment calculates similarity 

15 degrees manifested by the interpolation target pixel along 
two directions, i.e., a vertical direction and a 
horizontal direction, in which nearby pixels with color 
information corresponding to 'the first color component are 
connected to the interpolation target pixel and makes a 

20 judgment with regard to degrees of similarity manifested 
by the interpolation target pixel along the vertical 
direction and the horizontal direction based upon a 
difference between said similarity degrees; said second 
means for similarity judgment calculates similarity 

25 degrees manifested by the interpolation target pixel along 
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two diagonal directions in which nearby pixels with color 
information corresponding to the third color component are 
connected to the interpolation target pixel and makes a 
judgment with regard to degrees of similarity manifested 
5 by the interpolation target pixel along the two diagonal 
directions based upon a difference between said similarity 
degrees; and said means for interpolation value 
calculation selects color' information corresponding to the 
first color component or color information corresponding 
10 to the third color component to be used in conformance to 
the degrees of similarity judged by said second means for 
similarity judgment when calculating said third term. 
[0020] 

In the invention according to claim 5, the third 
15 color component is present at nearby pixels adjacent to 
the interpolation target pixel along the two diagonal, 
directions and the similarity judged by the second 
similarity judgment section, is the similarity manifested 
by the interpolation target pixel along the two diagonal 
20 directions.. 

Therefore, according to the invention of claim 5, the 
similarity along the diagonal directions can be reflected 
in the third term by switching between color information 
corresponding to the first color component and color 
25 information corresponding to the third color component to 
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be used in conformance to the degrees of similarity 
manifested by the interpolation target pixel along the two 
diagonal directions when calculating said third term. 
[0021] 

5 An interpolation processing apparatus according to 

claim 6 is achieved by that in the interpolation 
processing apparatus according to claim 5, said means for 
interpolation value calculation calculates the third term 
based upon color information corresponding to the first 

10 color component if said second means for similarity 

judgment judges that roughly equal degrees of similarity 
manifest along the two diagonal directions and calculates 
the third. term based upon color information corresponding 
to the third color component if' said second means for 

15 similarity judgment judges that a higher degree of 
similarity manifests along one of the two diagonal 
directions compared to the other diagonal direction. 
[0022] 

Namely, in the interpolation processing apparatus 
20 according to claim 6, the similarity manifesting along the 
diagonal directions is reflected with a high degree of 
reliability when calculating the third term. ^ 

An interpolation processing apparatus according to 
claim 7 is achieved by that in the interpolation 
25 processing apparatus according to claim 3: when image data. 
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are provided in a colorimetric system constituted of first 
- third color components with the first color component 
achieving a higher spatial frequency than the second color 
component and the third color component, the first color 
5 component set in a checker-board pattern, the second color 
component and the third color component each set in a line 
sequence between pixels at which color information 
corresponding to the first color component is present and 
information corresponding to the second color component 

10 present at the interpolation target pixel; said first 
means for similarity judgment calculates similarity 
degrees manifested by the interpolation target pixel along 
two directions, i.e., a vertical direction and a 
horizontal direction, in which nearby pixels with color 

15 , information corresponding to the first color component are 
connected to the interpolation target pixel and makes a . 
judgment with regard to degrees of similarity manifested 
by the interpolation target pixel along the vertical 
direction and the horizontal direction based upon a 

20 difference between said similarity degrees; said second 
means for similarity judgment calculates similarity 
degrees manifested by the interpolation target pixel along 
two diagonal directions in which nearby pixels with color 
information corresponding to the third color component are 

25 connected to the interpolation target pixel and makes a 
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judgment with regard to degrees of similarity manifested 
by the interpolation . target pixel along the two diagonal 
directions based upon a difference between said similarity 
degrees; and said means for interpolation value 
5 calculation selects color information corresponding to the 
first color component or color information corresponding 
to the third color component to be used in conformance to 
the degrees of similarity judged by said second means for 
similarity judgment when calculating said second term. 
10 [0023] 

In the invention according to claim 1, the third 
color component is present at nearby pixels adjacent to 
the interpolation target pixel along the two diagonal 
directions and the similarity judged by the second 
15 similarity judgment section is the similarity manifested 
by the interpolation target pixel along the two diagonal 
directions. 

Therefore, according to the invention of claim 7, the 
similarity along the diagonal directions can be reflected 

20 in the second term by switching between color information 
corresponding to the first color component and color 
information corresponding to the third color component to 
be used in conformance to the degrees of similarity 
manifested by the interpolation target pixel along the two 

25 diagonal directions when calculating said second term. 
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[0024] 

An interpolation processing apparatus according to 
claim 8 is achieved by that in the interpolation 
processing apparatus according to claim 7, said means for 
5 interpolation value calculation calculates the second term 
based upon color information corresponding to the first 
color component if said second means for similarity 
judgment judges that roughly equal degrees of similarity 
manifest along the two diagonal directions and calculates 

10 the second based upon color information corresponding to 
the third color component if said second means for 
similarity judgment judges that a higher degree of 
similarity manifests along one of the two diagonal 
directions compared to the other diagonal direction. 

15 [0025] 

Namely, in the interpolation processing apparatus 
according to claim 8, the similarity manifesting along the 
diagonal directions is reflected with a high degree of 
reliability when calculating the second term. 

20 An interpolation processing apparatus according to 

claim 9 is achieved by that in the interpolation 
processing apparatus according to claim 5 or 7 : said first 
means for similarity judgment judges that roughly equal 
degrees of similarity manifest along the vertical 

25 direction and the horizontal direction if a difference 
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between the similarity degrees along the vertical 
direction and the horizontal direction is smaller^ than a 
specific threshold value; and said second means for 
similarity judgment judges that roughly equal degrees of 
similarity manifest along the two diagonal directions if a 
difference between the similarity degrees along the two 
diagonal directions is smaller than a specific threshold 
value. 
[0026] 

As a result, in the invention according to claim 9, 
the adverse effect of noise can be reduced in the judgment 
of the similarity along the two directions, i.e., the 
vertical direction and the horizontal direction and the 
similarity manifesting along the two diagonal directions. 

An interpolation processing apparatus according to 
claim 10 is achieved by that in the interpolation 
processing apparatus according to claim 5 or 7: said first 
means for similarity judgment calculates the similarity 
degrees along the vertical direction and the horizontal 
direction by using color information corresponding to a 
plurality of color components for a single interpolation 
target pixel; and said second means for similarity 
judgment calculates the similarity degrees along the two 
diagonal directions by using color information- 
corresponding to a plurality of color components for a 
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single interpolation target pixel. 
[0027] 

In other words, in the invention according to claim 
10, color ' information corresponding to a plurality of 
5 color components is reflected in the judgment of the 

similarity manifesting along the vertical and horizontal 
directions and the similarity manifesting along the two 
diagonal directions. 

An interpolation processing apparatus according to 

10 claim 11 is achieved by that in the interpolation 

processing apparatus according to claim 10 said second 
means for similarity judgment calculates a similarity 
degree manifesting along each of the two diagonal 
directions through weighted addition of: 

15 (1) a similarity degree . component constituted of color 
information corresponding to the first color component 
alone; 

(2) a similarity degree component constituted of color 
information corresponding to the second color component 

20 alone; 

(3) a similarity degree , component constituted of color 
information corresponding to the third color component 
alone; and 

(4) a similarity degree component constituted of color 
25 information corresponding to the second color component 
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and the third color component. 
[0028] 

In other words, in the invention according to claim 
11, color information corresponding to a plurality of 
5 color components is reflected in the judgment of the 

similarity manifesting along the two diagonal directions. 

An interpolation processing apparatus according to 
claim 12 is achieved by that in the interpolation 
processing apparatus according to claim 5 or 7 : said first 

10 means for similarity judgment calculates similarity 

degrees along the vertical direction and the horizontal 
direction for each pixel and makes a judgment on 
similarity manifested by the interpolation target pixel 
along the vertical direction and the horizontal direction . 

15 based upon differences in similarity degrees manifesting 
at nearby pixels as well as at the interpolation target 
pixel; and said second means for similarity judgment 
calculates similarity degrees along the two diagonal . 
directions for each pixel and makes a judgment on 

20 similarity manifested by the interpolation target pixel 
along the .two diagonal directions based upon differences 
in similarity degrees manifesting at nearby pixels as well 
as at the interpolation target pixel. 
[0029] 

25 In other words, in the invention according to claim 
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12, color information corresponding to a plurality of 
color components is reflected in the judgment of the 
similarity manifesting along the vertical and horizontal 
directions and the similarity manifesting along the two 
5 diagonal directions. 

An interpolation processing apparatus according to 
claim 13 engages in processing on image data which are 
provided in a colorimetric system constituted of first 
nth (n ^ 3) color components and include color information 

10 corresponding to a single color component provided at each 
pixel to determine an interpolation value equivalent to 
color information corresponding to the first color 
component for a pixel at which the first color component 
is missing, and comprises: a means for first term 

15 calculation that calculates a first term representing 
average information of the first color component with 
regard to an interpolation target pixel to undergo 
interpolation processing by using "color information* 
corresponding to color components at nearby pixels set 

20 nearby the interpolation target pixel; a means for second 
term calculation that calculates a second term 
representing local curvature information with regard to a 
color component matching the color component at the 
interpolation target pixel by using color information 

25 corresponding to color components at pixels each 
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containing a color component same as the interpolation 
target pixel; and a means for interpolation value 
calculation that calculates an interpolation value by 
adding said second term multiplied by a weighting 
coefficient constituted of color information corresponding 
to a plurality of color components at nearby pixels and 
the interpolation target pixel to said first term. 
[0030] 

In other words, in the invention according to claim 
13, the interpolation value is calculated by correcting 
the- "average information of the first color component" 
with the "curvature information based upon a color 
component matching the color component at the 
interpolation target pixel" multiplied by a weighting 
coefficient constituted of color information corresponding 
to a plurality of color components present at the 
interpolation target pixel and nearby pixels. 

An interpolation processing apparatus according to 
claim 14 is achieved by that in the interpolation 
processing apparatus according to claim 13, said means for 
interpolation value calculation uses color information 
corresponding to a plurality of color components provided 
at the interpolation target pixel and at a plurality of 
pixels set along a predetermined direction relative to the 
interpolation target pixel to ascertain inclinations 
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manifesting in color information corresponding to the 
individual color components along the direction and 
calculates said weighting coefficient in conformance to a 
correlation manifesting among the inclinations in the 
color information corresponding to the individual color 
components. 
[0031] 

As a result, in the invention according to claim 14, 
the ''average information of the first color component" is 
corrected with the "curvature information based upon a 
color component matching the color component at the 
interpolation target pixel" multiplied by the weighting 
coefficient, and the weighting coefficient is calculated 
in conformance to the correlation among the inclinations 
of the color information corresponding to the different 
color components in the area nearby the interpolation 
target pixel. 
[0032] 

An interpolation processing apparatus according to 
claim 15 implements processing for supplementing a color 
component value at a pixel at which information 
corresponding to a color component is missing in image 
data provided in a colorimetric system constituted of a 
luminance component and the color component, with the 
luminance component having a higher spatial frequency than 
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the color component and the luminance component present 
both at pixels having information corresponding to the 
color component and at pixels lacking information 
corresponding to the color component, and comprises: a 
5 means for hue value calculation that calculates hue values 
at a plurality of pixels located near an interpolation 
target pixel to undergo interpolation processing and 
having both the luminance component and the color 
■component by using luminance component values and color 

10 component values at the individual pixels; a means for hue 
value interpolation that calculates a hue value at the 
interpolation target pixel by using a median of the hue 
values at the plurality of pixels calculated by said means 
for hue value calculation; and a means for color 

15 conversion that interpolates a color component at the 
interpolation target pixel by using the luminance 
component at the interpolation target pixel to convert the 
hue value at the interpolation target pixel calculated by 
said means for hue value interpolation to a color 

20. component . 
[0033] 

Namely, in. the invention according to claim 15, the 
hue value of the interpolation target pixel is calculated 
by using the median of the hue values of a plurality of 
25 pixels located near the interpolation target pixel. 
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An interpolation processing apparatus according to 
claim 16 implements processing for supplementing a 
luminance component at a pixel at which information 
corresponding to a luminance component is missing and 
supplementing a color component at a pixel at which 
information corresponding to a color component is missing, 
on image data provided in a colorimetric system 
constituted of the luminance component and the color 
component, with the luminance component having a higher 
spatial frequency than the color component and a given 
pixel having only information corresponding to either the 
luminance component or the color component, and comprises: 
a means for luminance component interpolation that 
interpolates a luminance component at a luminance 
component interpolation target pixel to^ undergo luminance 
component interpolation processing by using at least 
either "similarity manifesting between the luminance 
component -interpolation target pixel and a pixel near the 
luminance component interpolation target pixel" or "a 
plurality of color components within a local area 
containing the luminance component interpolation target 
pixel"; a means for hue value calculation that calculates 
hue values at a plurality of pixels located near an 
interpolation target pixel to undergo color component 
interpolation processing, having color component values 
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and having luminance component values interpolated by said 
means for luminance component interpolation, by using the 
luminance component values and color component values at 
the individual pixels; a means for hue value interpolation 
that calculates a hue value for the interpolation target 
pixel by using a median of the hue values at the plurality 
of pixels calculated by said means for hue value 
calculation; and a means for color conversion that 
interpolates a color component value for the interpolation 
target pixel by using the luminance component value at the 
interpolation target pixel to convert the hue value at the 
interpolation target pixel calculated by said means for 
hue value interpolation to a color component value. 
[0034] 

Namely, in the invention according to claim 16, the 
hue value of the interpolation target pixel is calculated 
by using the median of the hue values of a plurality of 
pixels located near the interpolation target pixel. 

An interpolation processing apparatus according to 
claim 17 is achieved by that in the interpolation 
processing apparatus according to claim 15 or 16, when the 
luminance component in the image data corresponds to a 
green color component and the color component in the image 
data corresponds to a red color component and a blue color 
component, said means for hue value interpolation 



Translation of 
JPll-363007 



calculates a hue value for the interpolation target pixel 
by using a median of hue values containing the red color 
component at pixels near the interpolation target pixel if 
the green color component is present but the red color 
5 component is missing at the interpolation target pixel and 
calculates a hue value for the interpolation target pixel 
by using a median of hue values containing the blue color 
component at pixels near the interpolation target pixel if 
the green color component is present but the blue color 
10 component is missing at the interpolation target pixel. 
[0035] 

In other words, in the invention according to claim 
11 f the hue value of the interpolation target pixel at 
which the green color component is present but the red 

15 color component is missing is calculated by using the 
median of the hue values containing the red color 
component from pixels located near the interpolation 
target pixel, whereas the hue value of the interpolation 
target pixel at which the green color component is present 

20 but the blue color component is missing is calculated by 
using the median of the hue values containing the blue 
color component from pixels located near the interpolation 
target pixel, 
[0036] 

25 An interpolation processing apparatus according to 
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claim 18 is achieved by that in the interpolation 
processing apparatus according to claim 15 or 16, when the 
luminance component in the image data corresponds to a 
green color component and the color component in the image 
data corresponds to a red color component and a blue color 
component, said means for hue value interpolation 
calculates a hue value for the interpolation target pixel 
by using a median of hue values containing the red color 
component at pixels set near the interpolation target 
pixel if the blue color component is present but the red 
color component is missing at the interpolation target 
pixel. 
[0037] , 

Namely, in the invention according to claim 18, the 
hue value of the interpolation target pixel at which the 
blue color component, is present but the red color 
component is missing is calculated by using the median of 
the hue values containing the red color component from 
pixels located near the interpolation target pixel. 
[0038] 

An interpolation processing apparatus according to 
claim 19 is achieved by that in the interpolation 
processing apparatus according to claim 15 or 16, when the 
luminance component in the image data corresponds to a 
green color component and the. color component in the image 
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data corresponds to a red color component and a blue color 
component, said means for hue value interpolation 
calculates a hue value for the interpolation target pixel 
by using a median of hue values containing the blue color 
5 component at pixels set near the interpolation target 

pixel if the red color component is present but the blue 
color component is missing at the interpolation target 
pixel . 
[0039] 

10 Namely, in the invention according to claim 19, the 

hue value of the interpolation target pixel at which the 
red color component is present but the blue color 
component is missing is calculated by using the median of 
the hue values containing the blue color component from 

15 pixels located near the interpolation target pixel. 

An interpolation processing apparatus according to 
claim 20, in the interpolation processing apparatus 
according to any one of claims 15 through 19: a color 
component missing at the interpolation target pixel 

20 presents at only one pixel among four pixels set 

symmetrically along the vertical direction and the 
horizontal direction; and said means for hue value 
interpolation comprises a first hue value interpolation 
unit that calculates a hue value for the interpolation 

25 target pixel by using a median of hue values at a 
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plurality of diagonally adjacent pixels if the hue values 
of the plurality of diagonally adjacent pixels adjacent to 
the interpolation target pixel along diagonal directions 
have been calculated by said means for hue value 
5 calculation, and a second hue value interpolation unit 
that calculates a hue value for the interpolation target 
pixel by using a median of hue values at a plurality of 
vertically and horizontally adjacent pixels if the hue 
values of the plurality of vertically and horizontally 
10 adjacent pixels adjacent to the interpolation target pixel 
in the vertical direction and the horizontal direction 
have been calculated by said means for hue value 
calculation or said first hue value interpolation unit. 
[0040] 

15 In other words, in the invention according to claim 

20, if the hue values at pixels adjacent along the 
diagonal directions are already calculated, the hue value 
of the interpolation target pixel is calculated by using' 
the median of the hue values at the diagonally adjacent 

20 pixels, and whereas if the hue values at pixels adjacent 
in the vertical and horizontal directions are already 
calculated, the hue value of the interpolation target 
pixel is calculated by using the median of the hue values 
at the vertically and horizontally adjacent pixels. 

25 A recording medium according to claim 21 recordes 
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therein an interpolation processing program to -implement 
on a computer processing for determining an interpolation 
value equivalent to color information corresponding to a 
first color component missing at a pixel, on image data 
provided in a colorimetric system constituted of first 
nth (n ^ 3) color components with color information 
corresponding to a single color component present at each 
pixel. The interpolation processing program comprises: an 
interpolation value calculation step in which an 
interpolation value including, at least 

(1) a first term indicating local average information 
with regard to the first color component 

(2) a second term indicating curvature information with 
regard to a color component matching a color component at 
the interpolation target pixel and 

(3) a third term indicating curvature information with 
regard to a color component other than the color component 
at the interpolation target pixel. 

[0041] 

Namely, the invention according to claim 21 
calculates an interpolation value by correcting the "first 
term indicating local average information with regard to 
the first color ^component" with the "second term 
indicating curvature information with regard to a color 
component matching a color component at the interpolation 
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target pixel" and the "third term indicating curvature 
information with regard to a color component other than 
the color component at the interpolation target pixel". 

A recording medium according to claim 22 records ' 
therein an interpolation processing program to implement 
on a computer processing for determining an interpolation 
value equivalent to color information corresponding to a 
first color component missing at a pixel, on image data 
provided in a colorimetric system constituted of first 
nth (n ^ 3 ) color components with color information 
corresponding to a single color component present at each 
pixel. The interpolation processing program comprises: an 
interpolation value calculation step in which an 
interpolation value- including at least two terms, i.e., a 
first term and a second term is calculated by using color 
information at nearby pixels set nearby an interpolation 
target pixel to undergo interpolation processing; a first 
similarity judgment step in which degrees of similarity to 
the interpolation target pixel are judged along at least 
two directions in which nearby pixels having color 
information corresponding to the first color component are 
connected with the interpolation target pixel; .and a 
second similarity judgment step in which degrees of 
similarity to the interpolation target pixel are judged 
along at least two directions other than the directions 
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along which the degrees of similarity are judged in said 
first similarity judgment step, and in said interpolation 
value calculation step, a direction in which pixels having 
color information to be used to calculate said first term 
5 are set is selected based upon results of a judgment made 
in said first similarity judgment step and a direction in 
which pixels having color information to be used to 
calculate said second term are set is selected based upon 
results of a judgment made in said second similarity 
10 judgment step. 
[0042] 

Namely, in the invention according to claim 22 since 
directions along which the second similarity judgment step 
judges degrees of similarity are different from directions 

15 along which the first similarity judgment step judges 
degrees of similarity, a direction along which color, 
information to be used to calculate the first term exists 
does not agree with a direction along which color 
information to be used to calculate the second term exists. 

20 Therefore, according to the invention of claim 22 the 

interpolation value can be calculated by using color 
information at nearby pixels located along a plurality of 
directions. Also, according to the invention of claim 22 
the first term and the second term can be calculated by 

25 using color information at nearby pixels set along the 
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direction in which a high degree of similarity is 
manifested or through weighted synthesis of color 
information from nearby pixels located along a plurality 
of directions, which is performed in correspondence to 
varying degrees of similarity. 
[0043] 

A recording medium according to claim 23 records 
therein an interpolation processing program to implement 
on a computer processing for determining an interpolation 
value equivalent to color information corresponding to a 
first color component missing at a pixel, on image data 
provided in a colorimetric system constituted of first 
nth (n ^ 3) color components with color information 
corresponding to a single color component present at each 
pixel. The interpolation processing program comprises: a 
first term calculation step in which a first term 
representing average information of the first color 
component with regard to an interpolation target pixel to 
undergo interpolation processing is calculated by using 
color information corresponding to a color component at 
nearby pixels set nearby the interpolation target pixel; a 
second term calculation step in which a second term 
representing local curvature information with regard to a 
color component matching the color component at the 
interpolation target pixel is calculated by using color 
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information corresponding to a color component at pixels 
each containing a color component same as the 
interpolation target pixel; and an interpolation ' value 
calculation step in which an interpolation value is 
5 calculated by adding said second term multiplied by a 
weighting coefficient constituted of color information 
corresponding to a plurality of color components provided 
at nearby pixels and the interpolation target pixel to the 
first term. 
10 [0044] 

In other words,, in the invention according to claim 
23, the interpolation value is calculated by correcting 
the "average information of the first color component" 
with the "curvature information based upon a color 

15 component matching the color component at the 

interpolation target pixel" multiplied by a weighting 
coefficient constituted of color information corresponding 
to a plurality of color components present at the 
interpolation target pixel and nearby pixels. 

20 A recording medium according to claim 24 records 

therein an interpolation processing program for 
implementing on a computer processing supplementing a 
color component value at a pixel at which information 
corresponding to a color component is missing, on image 

25 data provided in a colorimetric system constituted of a 

65 



Translation of 
JPll-363007 



luminance component and the color component, with the 
luminance component having a higher spatial frequency than 
the color component and the luminance component present 
both at pixels having information corresponding to the 
5 color component and at pixels lacking information 

corresponding to the color component. The interpolation 
processing program comprises: a hue value calculation step 
in which hue values for a plurality of pixels near an 
interpolation target pixel to undergo interpolation 

10 processing and having information corresponding 'to both 
the luminance component and the color component are 
calculated by using luminance component values and color 
component values at the individual pixels; a hue value 
interpolation step in which a hue value for the 

15 interpolation target pixel is calculated by using a median 
of the hue values at the plurality of pixels calculated in 
the hue value calculation step; and a color conversion 
step in. which a color component value at the interpolation 
target pixel is interpolated by using a value indicated by 

20 the luminance component present at the interpolation 

target pixel to convert the hue value of the interpolation 
target pixel calculated in the hue value interpolation 
step to a color component value. 
[0045] 

25 Namely, in the invention according to claim 24, the 
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hue value of the interpolation target pixel is calculated 
by using the median of the hue values of a plurality of 
pixels located near the interpolation target pixel. 

A recording medium according to claim 25 records 
therein an interpolation processing program for 
implementing on a computer processing for supplementing a 
luminance component value at a pixel at which information 
corresponding to a luminance component is missing and a 
color component value at a pixel at which information 
corresponding to a color component missing, on image data 
provided in a colorimetric system constituted of the 
luminance component and the color component, with the 
luminance component having a higher spatial frequency than 
the color component and information corresponding to 
either the luminance component or the color component 
present at each pixel. The interpolation processing 
program' comprises: a luminance component interpolation 
step in which a luminance component value is interpolated 
for a luminance component interpolation target pixel to 
undergo luminance component interpolation processing by 
using at least either "similarity between the luminance 
component interpolation target pixel and a pixel near the 
luminance component interpolation target pixel" or 
"information corresponding to a plurality of color 
components within a local area containing the luminance 
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component interpolation target pixel"; a hue value 
calculation step in which hue values at a plurality of 
pixels located near an interpolation target pixel to 
undergo color component interpolation processing, having 
color component values and having luminance component 
values interpolated in said luminance component 
interpolation step are calculated by using the luminance 
component values and color component values at the 
individual pixels; a hue value interpolation step in which 
a hue value for the interpolation target pixel is 
calculated by using a median of the hue values at the 
plurality of pixels calculated in the hue value 
calculation step; and a color conversion step in which a 
color component value is interpolated for the 
interpolation target pixel by using the luminance 
component value at the interpolation target pixel to 
convert the hue value at the interpolation target pixel 
calculated in said hue value interpolation step' to a color 
component value. 

[0046] ' . 

Namely, in the invention according to claim 25, the 
hue value of the interpolation target pixel is calculated 
by using the median of the hue values of a plurality of 
pixels located near the interpolation target pixel. 

Inventions [1] through [15] related to the above will 
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be disclosed as follows. 

[1] In the recording medium according to claim 22 having 
recorded therein an interpolation processing program, the 
interpolation processing program further comprises: a 
5 first similarity judgment step that judges degrees of 
similarity to the interpolation target pixel along at 
least two directions in which nearby pixels with color 
information corresponding to the first color component are 
connected with the interpolation target pixel; and a 

10 second step for similarity judgment that judges degrees of 
similarity to the interpolation target pixel along at 
least two directions other than the directions in which 
the degrees of similarity are judged by said first step 
for similarity judgment. The interpolation value 

15 calculation step selects a direction along which nearby 
pixels having color information to be used to calculate 
said first term and said second term are set based upon 
results of a judgment made' by said first similarity 
judgment step; and said interpolation value calculation 

20 step selects a direction along which nearby pixels having 
color information to be used to calculate said third term 
are set based upon results of a judgment made by said 
second similarity judgment step. 
[0047] 

25 [2] In the recording medium according to claim 22 having 
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recorded therein an interpolation processing program, said 
interpolation value calculation step calculates a first 
term containing local average information with regard to 
the first color component and curvature information with 
regard to a color component matching a color component at 
the interpolation target pixel, and a second term 
containing curvature information with regard to a color 
component other than the color component at the 
interpolation target pixel to calculate the interpolation 
value . 
[0048] 

[3] In the recording medium according to [1] having 
recorded therein an interpolation processing program: when 
image data are provided in a colorimetric system 
constituted of first third color components with the 
-first color component achieving a higher spatial frequency 
than the second color component and the third color 
component, the first color component set ' in a checker- 
board pattern, the second color component and the third 
color component each set in a line sequence between pixels 
at which color information corresponding to the first 
color component is present and information corresponding 
to the second color component present -at the interpolation 
target pixel; said first similarity judgment step 
calculates similarity degrees manifested by the 
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interpolation target pixel along two directions, i.e., a 
vertical direction and a horizontal direction, in which 
nearby pixels with color information corresponding to the 
first color component are connected to the interpolation 
5 target pixel and makes a judgment with regard to degrees 
of similarity manifested by the interpolation target pixel 
along the vertical direction and the horizontal direction 
based upon a difference between said similarity degrees; 
said second similarity judgment step calculates similarity 

10 degrees manifested by the interpolation target pixel along 
two diagonal directions in which nearby pixels with color 
information corresponding to the third color component are 
connected to the interpolation target pixel and makes a 
judgment with regard to degrees of similarity manifested 

15 by the interpolation target pixel along the two diagonal 

directions based upon a difference between said similarity 
degrees; and said interpolation value calculation step 
selects color information corresponding to the first color 
component or color information corresponding to the third 

20 color component to be used in conformance to the degrees 
of similarity judged by said second similarity judgment 
step when calculating said third term. 
[0049] 

[4] In the recording medium according to [3] having 
25 recorded therein an interpolation processing program, said 
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interpolation value calculation step calculates the third 
term based upon color information corresponding to the 
first color component if said second similarity judgment 
step judges that roughly equal degrees of similarity 
5 manifest along the two diagonal directions and calculates 
the third term based upon color information corresponding 
to the third color component if said second similarity 
judgment step judges that a higher degree of similarity 
manifests along one of the two diagonal directions 
10 compared to the other diagonal direction. 
[0050] 

[5] In the recording medium according to claim 22 having 
recorded therein an interpolation processing program: when 
image data are provided in a colorimetric system 

15 constituted of first third color components with the 

first color component achieving a higher spatial frequency 
than the second color component and the third color 
component, the first color component set in a checker- 
board pattern, the second color component and the third 

20 color component each set in a line sequence between pixels 
at which color information corresponding to the first 
color component is present and information corresponding 
to the second color component present at the interpolation 
target pixel; said first similarity judgment step 

25 calculates similarity degrees manifested by the 
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interpolation target pixel along two directions, i.e.;- a 
vertical direction and a horizontal direction, in which 
nearby pixels with color information corresponding to the 
first color component are connected to the interpolation 
5 target pixel and makes a judgment with regard to degrees 
of similarity manifested by the interpolation target pixel 
along the vertical direction and the horizontal direction 
based upon a difference between said similarity degrees; 
said second similarity judgment step calculates similarity 

10 degrees manifested by the interpolation target pixel along 
two diagonal directions in which nearby pixels with color 
information corresponding to the third color component are 
connected to the interpolation target pixel and makes a 
judgment with regard to degrees of similarity manifested 

15 by the interpolation target pixel along the two diagonal 

directions based upon a difference between said similarity 
degrees; and said interpolation value calculation step 
selects color information corresponding to the first ' color 
component or color information corresponding to the third 

20 color component to be used in conformance to the degrees 
of similarity judged by said second similarity judgment 
step when calculating said second term. 
[0051] 

[6] In the recording medium according to [5] having 
25 recorded therein an interpolation processing program, said 
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interpolation value calculation step calculates the second 
term based upon color information corresponding to the. 
first color component if said second similarity judgment 
step judges that roughly equal degrees of similarity 
5 manifest along the two diagonal directions and calculates 
the second based upon color information corresponding to 
the third color component if said second similarity 
judgment step judges that a higher degree of similarity 
manifests along one of the two diagonal directions 
10 compared to the other diagonal direction. 
[0052] 

[7] In the recording medium according to [3] or [5] 
having recorded therein an interpolation processing 
program: said first similarity judgment step judges that 

15 roughly equal degrees of similarity manifest along the 
vertical direction and the horizontal direction if a 
difference between the similarity degrees along the 
vertical direction and the horizontal direction is smaller 
than a specific threshold value; and said second 

20 similarity judgment step judges that roughly equal degrees 
of similarity manifest along the two diagonal directions 
if a difference between the similarity degrees along the 
two diagonal directions is smaller than a specific 
threshold value. 

25 [0053] 
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[8] In the recording medium according to [3] or [5] 
having recorded therein an interpolation processing 
program: said first similarity judgment step calculates 
the similarity degrees along the vertical direction and 
5 the horizontal direction by using color information 

corresponding to a plurality of color components for a 
single interpolation target pixel; and said second 
similarity judgment step calculates the similarity degrees 
along the two diagonal directions by using color 
10 information corresponding to a plurality of color 

components for a single interpolation target pixel. 
[0054] 

[9] In the recording medium according to [8] having 
recorded therein an interpolation processing program, said 
15 second similarity judgment step calculates a similarity 
degree manifesting along each of the two diagonal 
directions through weighted addition of: 

(1) a similarity degree component constituted of color 
information corresponding to the first color component 

20 alone; 

(2) a similarity degree component constituted of color 
information corresponding to the second color component 
alone; 

(3) a similarity degree component constituted of color 
25 information corresponding to the third color component 
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alone; and 

(4) a similarity degree component constituted of color 
information corresponding to the second color component 
and the third color component. 
5 [0055] 

[10] In the recording medium according to [3] or [5] 
having recorded therein an interpolation processing 
program: said first similarity judgment step calculates 
similarity degrees along the vertical direction and the 

10 horizontal direction for each pixel and makes a judgment 

on similarity manifested by the interpolation target pixel 
along the vertical direction and the horizontal direction 
based upon differences in similarity degrees manifesting 
at nearby pixels as well as at the interpolation target 

15 pixel; and said second similarity judgment step calculates 
similarity degrees along the two diagonal directions for 
each pixel and makes a judgment on similarity manifested 
by "the interpolation target pixel along the two diagonal 
directions based upon differences in similarity degrees 

20 manifesting at nearby pixels as well as at the 
interpolation target pixel. 
[0056] ' 

[11] In the recording medium according to claim 23 having 
recorded therein an interpolation processing program said 
25 interpolation value calculation step uses color 
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information corresponding to a plurality of color 
components provided at the interpolation target pixel and 
at a plurality of pixels set along a predetermined 
direction relative to the interpolation target pixel to 
5 ascertain inclinations manifesting in color information 

corresponding to the individual color components along the 
direction and calculates said weighting coefficient in 
conformance to a correlation manifesting among the - 
inclinations in the color information corresponding to the 
10 individual color components. 
[0057] 

[12] In the recording medium according to claim 24 or 25 
having recorded therein an interpolation processing 
program, when the luminance component in the image data 

15 corresponds to a green color component and the color 

component in the image data corresponds to a red color 
component and a blue color component, said hue value 
interpolation step calculates a hue value for the 
interpolation target pixel by using a median of hue values 

20 containing the red color component at pixels near the 

interpolation target pixel if the green color component is 
present but the red color component is missing at the 
interpolation target pixel and calculates a hue value for 
the interpolation target pixel by using a median of hue 

25 values containing the blue color component at pixels near 
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the interpolation target pixel if the green color 
component is present but the blue color component is 
missing at the interpolation target pixel. 
[0058] 

5 [13] In the recording medium according to claims 24 or 25 
having recorded therein an interpolation processing 
program, when the luminance component in the image data 
corresponds to a green color component and the color 
component in the image data corresponds to a red color 

10 component and a blue color component, said hue value 
interpolation step calculates a hue value for the 
interpolation target pixel by using a median of hue values 
containing the red color component at pixels set near the 
interpolation target pixel if the blue color component is 

15 present but the red color component is missing at the 
interpolation target pixel. 
[0059] 

[14] In the recording medium according to claims "24 or 25 
having recorded therein an interpolation processing 

20 program, when the luminance component in the image data 
corresponds to a green color component and the color 
component in the image data corresponds to a red color 
component and a blue color component, said hue value 
interpolation step calculates a hue value for the 

25 interpolation target pixel by using a median of hue values 
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containing the blue color component at pixels set near the 
interpolation target pixel if the red color component is 
present but the blue color component is missing at the 
interpolation target pixel. 
5 [0050] 

[15] In the recording medium according to claim 24, claim 
25, [12], [13] or [15] having recorded therein an 
interpolation processing program, with a color component 
missing at the interpolation target pixel present at only 

10 one pixel among four pixels set symmetrically along the 

vertical direction and the horizontal direction, said hue 
value interpolation step comprises: a first hue value 
interpolation unit that calculates a hue value for the 
interpolation target pixel by using a median of hue values 

15 at a plurality of diagonally adjacent pixels if the hue 
values of the plurality of diagonally adjacent pixels 
adjacent to the interpolation target pixel along diagonal 
directions have been calculaited by said hue value 
calculation step; and a second hue value interpolation 

20 unit that calculates a hue value for 'the interpolation 
target pixel by using a median of hue values at a 
plurality of vertically and horizontally adjacent pixels 
if the hue values of the plurality of vertically and 
horizontally adjacent pixels adjacent to the interpolation 

25 , target pixel in the vertical direction and the horizontal 
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direction. have been calculated by said hue value 
calculation step or said first hue value interpolation 
unit . 
[0061] 

5 [Preferred Embodiment] 

The following is a detailed explanation of the 
embodiments of the present invention, given in reference 
to the drawings. FIG. 1 is a functional block diagram of 
the electronic camera corresponding to the first through 

10 fourth embodiments. 

It is to be noted that an electronic camera according 
to the first embodiment corresponds to an electronic 
camera comprising functions of interpolation processing 
executed by an interpolation processing apparatus 

15 according to any one of claims 1-12 and 15-20, an 

electronic camera according to the second embodiment 
corresponds to an electronic camera comprising functions 
of interpolation processing executed by an' interpolation 
processing apparatus according to any one of claims 1-4 

20 and 15-20, and an electronic camera according to the third 
embodiment or the fourth embodiment corresponds to an 
electronic camera comprising functions of interpolation 
processing executed by an interpolation processing 
apparatus according to any one of claims 13 and 14. 

25 [0062] 
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In FIG, 1, an electronic camera 10 comprises a 
control unit 11, a photographic optical system 12, an 
image-capturing unit 13, an A/D conversion unit 14, an 
image processing unit 15 and a recording unit 16. The 
image processing unit 15 is provided with an interpolation 
processing unit (e.g., a one-chip microprocessor dedicated 
to interpolation processing) . The image-capturing unit 13 
is provided with an image-capturing sensor (not shown) 
constituted by arranging R, G and B color filters in a. 
Bayer array. 
[0063] 

It is to be noted that while FIG. . 1 shows only the 
interpolation processing unit 17 in the image processing 
unit 15 to simplify the illustration, a functional block 
that engages in other image processing such as gradation 
conversion processing may also be provided in the image 
processing unit 15. 

In FIG. 1, the control unit 11 is 'connected to the 
image-capturing unit 13, the A/D conversion unit 14, the 
image processing unit 15 and the recording unit 16. In 
addition, an optical image obtained at the photographic 
optical system 12 is formed at the image-capturing sensor 
in the image-capturing unit 13. An output from the image- 
capturing unit 13 is quantized at the A/D conversion unit 
14 and is provided to the image processing unit 15 as 
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image data. The image data provided to the image 
processing unit 15 undergo" interpolation processing at the 
interpolation processing unit 17 and they are recorded via 
the recording unit 16.. 
5 [0064] 

FIG. 2 shows the arrangements of the color components 
in the image data adopted in the first embodiment and the 
third embodiment, and FIG. . 3shows the arrangements of the 
color components in the image data adopted in the second 

10 embodiment and the fourth embodiment. It is to.be noted 
that in FIGS. 2A and 2B and in FIGS. 3A and 3B, the 
individual color components are indicated as R, G and 
with the positions of pixels at which the various color 
components are present indicated with i and j . 

15 [0065] 

With [i,j] indicating the coordinates of an 
interpolation target pixel to undergo the interpolation 
processing, FIG. 2 shows the arrangement of 7 X 7 pixels 
with the interpolation target pixel at the center, whereas 

20 FIG. 3 shows the arrangement of 5 X 5 pixels with the 

interpolation target pixel at the center. In addition, 
FIGS. 2(1) and 3(1) each show an arrangement of pixels 
among which a pixel with the red color component is to 
undergo the interpolation processing, and FIGS. 2(2) and 

25 3(3) each show an arrangement of pixels among which a 
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pixel with the blue color component is to undergo the 

interpolation processing . 

[0066] 

In the various embodiments to be detailed later, the 
5 interpolation processing unit 17 first implements the 
interpolation processing to supplement the green color 
interpolation values for pixels at which the green color 
component is missing (hereafter referred to as "G 
interpolation processing") and then engages in 

10 interpolation processing through which red color 

interpolation values and blue color interpolation values 
are supplemented at pixels at which the red color 
component and the blue color component are missing 
(hereafter referred to as "RB interpolation processing"). 

15 However, since the interpolation processing implemented to 
supplement blue color interpolation values (hereafter 
referred to as ^'B interpolation processing") is 
implemented in a manner identical to the manner with which 
the interpolation processing for supplementing red color 

20 interpolation values (hereafter referred to as "R 
interpolation processing ") is implemented, its 
explanation is omitted. 
[0067] 

In addition, it is assumed that the pixel at 
25 coordinates [i,j] is the interpolation target pixel to 
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undergo the G interpolation processing, to simplify the 
subsequent explanation. Since the green color 
interpolation value can be calculated through the G 
interpolation processing in each of the embodiments 
5 explained below regardless of the color component (red or 
blue) at the interpolation target pixel, R and B in FIG. 2 
and FIG. 3 are all replaced with Z with the color 
information at the interpolation target pixel expressed as 
Z[i,j] and color information at other pixels also 
10 expressed in a similar manner in the following explanation. 
[0068] 

( First Embodiment ) 

FIGS. 4 and 5 present a flowchart of the operation 
achieved in the interpolation processing unit 17 in the 

15 first embodiment, with FIG. 4 corresponding to the 

operation of the interpolation processing unit 17 during 
the G interpolation processing and FIG. 5 corresponding to 
the operation of the interpolation processing unit 17 
during the R interpolation processing. 

20 The explanation of the operation achieved in the 

first embodiment given below focuses on the operation of 
the interpolation processing unit 17 by referring to FIGS. 
4 and 5. 
[0069] 

25 First, the interpolation processing unit 17 
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"calculates a similarity degree Cv[i,j] along the vertical 
direction and a similarity degree Ch[i,j] along the 
horizontal direction for an interpolation target pixel at 
which the green component is missing (FIG. 4 SI). 
5 Now, the details of the processing implemented to 

calculate the vertical similarity degree Cv[i,j] and the 
horizontal similarity degree Ch[i,j] in the first 
embodiment are explained. 
[0070] 

10 The interpolation processing unit 17 first calculates 

a plurality of types of similarity degree components along 
the vertical direction and the horizontal direction 
defined through the following formulae 10-21. 

G-G similarity degree component along vertical 
15 direction: Cvl [ i , j ] = | G [ i , j -1 ] -G [ i , j +1 ] | formula 10 
G-G similarity degree component along horizontal 
direction: Chi [i, j ] = | G [i-1, j ] -G [i + 1, j ] I formula 11 

B-B (R-R) similarity' degree component along vertical 
direction: Cv2 [i,j] = (|Z[i-l,j-l]-Z[i-l,j+l] | + | Z [i + 1, j-1] - 
20 Z [i+1, j+1] I ) /2 formula 12 

B-B (R-R) similarity degree component along 
horizontal direction: Ch2 [i,j]=(|Z[i-l,j-l]-Z[i+l,j- 
1] I + I Z [i-1, j+1] -Z [i + 1, j+1] I ) /2 formula 13 

R-R (B-B) similarity degree component along vertical 
25 direction: Cv3 [i, j ] = ( | Z [i, j-2] -Z [i, j ] | + |Z[i,j+2]- 
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Z [i, j ] I ) /2 formula 14 

R-R (B-B) similarity degree component along 
horizontal direction: Ch3[i,j]=( |Z[i-2,j]- 
Z [i, j ] I + I Z [i+2, j ] -Z [i, j ] I ) /2 ••• formula 15 
5 G-R (G-B) similarity degree component along vertical 

direction: Cv4[i, j]=( |G[i,j-l]-Z[i,j] |+|G[i,j+l]- 
Z[i,j]|)/2 formula 16 ' 

G-R (G-B) similarity degree component along 
horizontal direction: Ch4 [ i , j ] = ( I G [ i-1 , j ] - 
10 Z[i, j] | + |G[i + l, j]-Z[i, j] I )/2 ••• formula 17 

B-G (R-G) similarity degree component along vertical 
direction: Cv5[i, j]=( | Z [ i-1 , j -1 ] -G [ i-1 , j ] |+| Z[i-lv j+lJ- 
GLi-l^j] |+|Z[i+l, j-l]-G[i+l, j] |+|Z[i+l, j+l]-G[i+l, j] |)/4 
formula 18 

15 B-G (R-G) similarity degree component along 

horizontal direction: Ch5 [i,j]==( |Z[i-l,j-l]-G[i,j- 
1] |+|Z[i-l, j+l]-G[i, j+1] |+|Z[i+l, j-l]-G[i, j- 
. 1] I + I Z [i + 1, j+1] -G [i, j+1] I ) /4 formula"'19 

luminance similarity degree component along vertical 
20 direction: Cv6[i, j]={ |Y[i,j-l]-Y[i,j] |+|Y[i,j+l]- 
Y[i, j] I ) /2 formula 20 

luminance similarity degree component along 
horizontal direction : Ch6[i,j]=(|Y[i-l,j]- 
Y[i, j] l + l Y[i-fl, j]-Y[i, j] I )/2 "* formula 21 
25 In formulae 20 and 21, Y[i,j] represents a value 
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calculated through 

Y[i, j] = (4-A[i, j ]+2- (A[i, j-l]+A[i, j+l]+A[i- 
l, j]+A[i+l, j] )+A[i-l, j-l]+A[i-l, j+l]+A[i+l, j- 
l] +A[i+1, j+1] ) /16 formula 22 

, which is equivalent to the luminance value generated 
through filtering processing in which the color 
information corresponding to the color components at 
nearby pixels around the interpolation target pixel is 
averaged at a ratio of R:G:B = 1:2:1. It is to be noted 
that A[i, j] represents an arbitrary set of color 
information on the Bayer array which may assume a G value 
or a Z value depending upon the position at which the 
color information is provided. 
[0071] 

Next, the interpolation processing unit 17 performs 
weighted addition of the plurality of types of similarity 
degree components along each direction by using weighting 
coefficients al, a2, a3, BiA f a5 and a6, as expressed in 
the follow formulae 23 and 24. 

CvO [i, j ] - (al-Cvl [i, j ] +a2-Cv2 [i, j ] +a3-Cv3 [i, j ] +a4-Cv4 [i, j ] 
+a5*Cv5 [i, j ] +a6-Cv6 [i, j ] ) / (al+a2-fa3+a4+a5+a6) 
... formula 23 

ChO [i, j ] = (al-Chl [i, j ] +a2-Ch2 [i, j ] +a3-Ch3 [i, j ] +a4-Ch4 [i, j ] 
+a5-Ch5 [i, j ] +a6-Ch6 [i, j ] ) / (al+a2+a3+a4+a5+a6) 
... formula 24 
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It is to be noted that the ratio among the weighting 
coefficients al, a2, a3, a4, a5 and a6 in formulae 23 and 
24 may be, for instance, " al: a2 : a3: a4 : a5 : a6: = 
2:1:1:4:4:12 
5 ■ [0072] 

In the first embodiment, the similarity degrees are 
calculated by calculating the similarity degree components 
along the vertical and horizontal directions and 
performing weighted addition of the similarity degree 
10 components for nearby pixels around the interpolation 

target pixel as well as for the interpolation target pixel. 

The interpolation processing unit 17 performs 
weighted addition of the results obtained by implementing 
weighted addition of the similarity degree components at 
15 the interpolation target pixel and the nearby pixels 
(CvO [i, j ] , CvO [i-1, j-1] , CvO [i-1, j+1] , CvO [i+l, j-1] , 
CvO[i+l,j+l] and the like), through either (method 1) or 
(method 2) detailed below, to obtain a similarity degree 
Cv[i,j] along the vertical direction and a similarity 
20 degree Ch[i,j] along the horizontal direction manifesting 
by the interpolation target pixel. 
[0073] 
(method 1) 

Cv [i, j ] = (4-CvO [i, j ] +CvO [ i-1 , j -1 ] +CvO [i-1, j+1] 
25 +CvO [i + 1, j-1] +CvO [i + 1, j+1] ) /8 ... formula 25 
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Ch [i, j ] = (4-ChO [i, j ] +ChO [ i-1 , j -1 ] +ChO j+1] 

-fChO [i + 1, j-1] +ChO [i + 1, j+1] ) /8 ... formula 26 
(method 2) 

Cv[i, j] = (4-CvO [i, j] 
5 +2- (CvO [i-1, j-1] +CvO [i+1, j-1] +CvO [ i-1 , j +1 ] +CvO [i+1, j+1] ) 

+CvO [i, j-2] +CvO [i, j+2] +CvO [i-2, j ] +CvO [i+2, j ] ) /16 
... formula 27 
Ch[i, j]=(4-ChO[i, j] 

+2- (ChO [i-1, j-1] +ChO [i+1, j-1] +ChO [ i-1 , j +1 ] +ChO [i+1, j+1] ) 
10 +ChO [i, j-2] +ChO [i, j+2] +ChO [i-2, j ] +ChO [i+2, j ] ) /16 

... formula 28 

It is to be noted that while (method 1) corresponds 
to that weighted addition of the similarity degree 
components at the interpolation target pixel and the 

15 nearby pixels is implemented as illustrated in FIG., 6(1), 
(method 2) corresponds to that weighted addition of the 
similarity degree components at the interpolation target 
pixel and the nearby pixels is implemented as illustrated 
in FIG. 6(2). 

20 [0074] 

The similarity degree components each calculated by 
using color information corresponding to the same color 
component such as the G-G similarity degree components, B- 
B (R-R) similarity degree component and the R-R (B-B) 
25 similarity degree components (hereafter referred to as 
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"same-color similarity degree components") have been 
confirmed through testing to be suitable for use in the 
evaluation of similarity manifesting in an image with a 
low spatial frequency and a large colored area. The 
5 similarity degree components each calculated by using 
color information corresponding to different color 
components such as- the G-R (G-B) similarity degree 
components and B-G (R-G) similarity degree components 
(hereafter referred to as "different-color similarity 

10 degree components") have been confirmed through testing to 
be suitable for use in the evaluation of similarity 
manifesting in an image with a high spatial frequency and 
a large achromatic image area. In addition, the luminance 
similarity degree components have been confirmed through 

15 testing to be suitable for use in the evaluation of . 
similarity manifesting in an image containing both a 
colored area and an image area with a fairly high spatial 
frequency. ' ' . 

[0075] 

20 In other words, the evaluation of similarity 

manifesting in various types of images can be achieved 
with a high degree of accuracy by using similarity degrees 
obtained through weighted addition of same-color 
similarity degree components, different-color similarity 

25 degree components and luminance similarity degree 
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components . 

In addition, the functions of the three types of 
similarity degree components calculated as the. same-color 
similarity degree components (the G-G similarity degree 
5 components, the B-B (R-R) similarity degree components and 
the R-R (B-B) -similarity degree components) in the 
similarity evaluation can be complemented by one another 
and the functions of the two types of. similarity degrees 
components calculated as the different-color similarity 
10 degree components (the G-R (G-B) similarity degree 

components and the B-G (R-G) similarity degree components) 
in the similarity evaluation, too, can be complemented by 
each other . 
[0076] 

15 Furthermore, in the first embodiment, the vertical 

similarity degree Cv[i,j] and the horizontal similarity 
degree Ch[i,j] are calculated through weighted addition of 
the results of weighted addition of similarity degree 
components at the interpolation target pixel and the 

20 results of weighted addition of similarity degree 

components at nearby pixels. Thus, the continuity between 
the color information at the interpolation target pixel 
and the color information at the pixels located near the 
interpolation target pixel is readily reflected in the 

25 vertical similarity degree Cv[i,j] and the horizontal 
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similarity degree Ch[i,j]. 
[0077] 

In particular , the vertical similarity degree 
Cv[i,j] and the horizontal similarity degree Ch[i,j] 
calculated through (method 2) reflect color information 
corresponding to the color components at pixels over a 
wide range and thus, are effective in the similarity 
evaluation of an image manifesting a pronounced 
magnification chromatic aberration. 

It is to be noted that the vertical similarity degree 
Cv[i,j] and the horizontal similarity degree Ch[i,j] in 
the first embodiment indicate more marked similarity as 
their values become smaller. 
[0078] 

After the vertical similarity degree Cv[i,j] and the 
horizontal similarity degree Ch[i,j] are calculated as 
described above, the interpolation processing unit 17 
compares the similarity along the vertical direction and 
the similarity along the horizontal direction manifesting 
at the interpolation target pixel (hereafter referred to 
as the "vertical/horizontal similarity") based upon the 
vertical similarity degree Cv[i,j] and the horizontal 
similarity degree Ch[i,j] (FIG. 4 S2). Then, it sets one 
of the following values for an index HV[i,j] which 
indicates the vertical/horizontal similarity based upon 
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the results of the comparison. 
[0079] 

For instance, if |Cv[i,j] - Ch[i,j]| > Tl and Cv[i,j] 
< Ch[i,j] are true with regard to a given threshold value 
5 Tl, the interpolation processing unit 17 judges that a 
more marked similarity is manifested along the vertical 
direction than along the horizontal direction and sets 1 
for the index HV[i,j] (FIG. 4 S3), if; |Cv[i,j] - Ch[i,j]| 
> Tl and Cv[i,j] > Ch[i,j] are true, the interpolation 

10 processing unit 17 judges that a more marked similarity is 
manifested along the horizontal direction than along the 
vertical direction and sets -1 for the index HV[i,j] (FIG. 
4 S4) and if; |Cv[i,j] - Ch[i,j]| ^ Tl is true, the 
interpolation processing unit 17 judges that the degree of 

15 similarity manifested along the horizontal direction and 
along the vertical direction are essentially the same and 
sets 0 for the index HV[i,j] (FIG. 4 S5) . 
[0080] 

It is to be noted that the threshold value Tl is used 
20 to prevent an erroneous judgment that the similarity along 
either direction is more marked from.being made due to 
noise when the difference between the vertical similarity 
degree Cv[i,j] and the horizontal similarity degree 
Ch[i,j] is very little. Accordingly^ by setting a high 
25 value for the threshold value Tl when processing a color 
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image with a great deal of noise, an improvement in the 
accuracy of the vertical/horizontal similarity judgment is 
achieved. 

Next, the interpolation processing unit 17 calculates 
5 a similarity degree C45[i,j] along the diagonal 45° 
direction and a similarity degree C135[i,j] along the 
diagonal 135° degree direction for the interpolation 
target pixel (FIG. 4 S5) . 
[0081] 

10 Now, details of the processing implemented in the 

first embodiment to calculate the diagonal 45° similarity 
degree C45[i,j] and the diagonal 135° similarity degree 
C,135[i,j] are explained. 

First, the interpolation processing unit 17 

15 calculates a plurality of types of similarity degree 
components along the diagonal 45° direction and the 
diagonal 135° direction as defined in the following 
formulae 2 9 36; 

G-G similarity degree component along the diagonal 45° 
20 direction: 

C45_l[i, j]=( |G[i, j-l]-G[i-l, j] |+|G[i+l, j]-G[i, j+1] I )/2 
... formula 29 

G-G similarity degree component along the diagonal 135° 
direction: 

25 C135_l[i, j]=( |G[i, j-l]-G[i+l, j] |+|G[i-l, j]-G[i, j+1] I ) /2 
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... formula 3 0 

B-B (R-R) similarity degree component along the diagonal 
45° direction: 

C45_2[i, j] = |Z[i + l, j-l]-Z[i-l, j + 1] I ... formula 31 

B-B (R-R) similarity degree component along the diagonal 

135"* direction: 

C135_2[i, j] = |Z[i-l, j-l]-Z[i + l, j-fl] I ... formula 32 

R-R (B-B) similarity degree component along the diagonal 
45° direction: 

C4 5_3[i, j]=( |Z[i+2, j-2]-Z[i, j] |+|Z[i-2, j+2]-Z[i, j] | ) /2 
... formula 33 

R-R (B-B) similarity degree component along the diagonal 
135° direction: 

C135_3[i, j]=(|Z[i-2, j-2]-Z[i, j] 1+ | Z [i+2, j+2] -Z [i, j ] |) /2 
... formula 34 ; 

B-R (R-B) similarity degree component along the diagonal 
45° direction: 

C4 5_4 [i, j] = ( I Z [i + 1, j-1] -Z [i, j] 1 + I Z [i-1, j+1] -Z [i, j ] I ) /2 
... formula 35 

B-R (R-B) similarity degree component along the diagonal 
135° direction: 

C135_4 [i, j ]=(| Z [i-1, j-l]-Z [i, j] l+l Z [i+1, j+1] -Z [i, j] I ) /2 
... formula 36; 

Next, the interpolation processing unit 17 performs 
weighted addition of the plurality of types of similarity 
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degree components performed along each of the two 
directions by using weighting coefficients bl, b2, b3 and 
b4 , as expressed in the following formulae 37 and 38. 
[0082] 

5 C4 5_0 [i, j ] = (bl-C4 5_l [i, j ] +b2-C4 5_2 [i, j ] +b3-C4 5_3 [i, j ] 
+b4-C45_4 [i, j ] ) / (bl+b2+b3-fb4) ... formula 37 
C135_0 [i, (bl-C135_l [i, j ] +b2-C135_2 [i, j ] +b3-C135_3 [i, j ] 
+b4-C135_4 [i, j ] ) / (bl+b2+b3+b4) ... formula 38 
It is to be noted that the ratio of the weighting 
10 coefficients bl, b2, b3 and b4 in formulae 37 and 38 may 
be, for instance, "bl : b2 : b3 : b4 : - 2:1:1:2." 
[0083] 

In the first embodiment, a further improvement is 
achieved in the accuracy with which the similarity degrees 

15 are calculated by calculating the similarity degree 
components along the diagonal 45° direction and the 
diagonal 135° direction and performing weighted addition 
of the similarity degree components for nearby pixels 
around the interpolation target pixel as well as for the 

20 interpolation target pixel. 

Namely, the interpolation processing unit 17 performs 
weighted addition of the results obtained by implementing 
weighted addition of the similarity degree components at 
the interpolation target pixel and the nearby pixels 

25 (C45_0[i,j], C45_0 [i-1, j-1] , C45_0 [i-1, j-fl] , C45_0[i + l,j- 
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1], C45_0 [i+1, j +1 ] and the like) through either (method 1) 
or (method 2) detailed below, to obtain a similarity 
degree Cv[i,j] along the diagonal 45° direction and a 
similarity degree Ch[i,j] along the diagonal 135° 
direction manifesting by the interpolation target pixel 
(equivalent to implementing weighted addition of 
similarity degree components at the interpolation target 
pixel and the nearby pixels as illustrated in FIGS. 6(1) 
and 6B) . 
[0084] 
(method 1) 

C4 5 [i, j ]= (4-C4 5_0 [i, j ] +C4 5_0 [ i-1 , j -1 ] +C4 5_0 [i+1, j-1] 
+C45_0 [i-1, j+1] +C45_0 [i + 1, j+1] ) /8 ... formula 39 

C135 [i, j ] = (4-C135__0 [i, j ] +C135_0 [ i-1 , j -1 ] +C135_0 [i + 1, j-1] 
+C135_0 [i-1, j + 1] +C135_0 [i + 1, j+1] ) /8 ... formula 40 

(method 2) 

C4 5[i, j]=(4-C4 5_0 [i, j] +2- (C4 5_0 [i-1, j -1] +C4 5_0 [i+1, j-1] 
+C4 5_0 [i-1, j+1] +C4 5__0 [i + 1, j+1] ) +C4 5_0 [i, j-2] 
+C4 5_0 [i, j+2] +C4 5_0 [i-2, j ] +C4 5_0 [i+2, j ] ) /16 
... formula 41 

C135[i, j]=(4'C135_0 [i, j]+2- ( C135_0 [ i-1 , j -1 ] +C135_0 [i+1, j-1] 
+C135_0 [i-1, j+1] +C135_0 [i+1. j+1] ) +C135_0 [i, j -2] 
+C135_0 [i, j+2] +C135_0 [i-2, j ] +C135_0 [i+2, j ] ) /16 
... formula 42 

It is to be noted that the weighted addition of the 
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plurality of similarity degree components and the 
consideration of the evaluation of similarity degrees at 
the nearby pixels with regard to the diagonal 45° 
similarity degree C45[i,j] and the diagonal 135° 
5 similarity degree C135[i,j] thus calculated achiev.es the 
same function as that with regard to the vertical 
similarity degree Cv[i,j] and the horizontal similarity 
degree Ch[i,j]. In addition, the diagonal 45° similarity 
degree C45[i,j] and the diagonal 135° similarity degree 
10 C135[i,j] in the first embodiment indicate more marked 
similarity as their values become smaller. 
[0085] 

After the diagonal 45° similarity degree C45[i,j] and 
the diagonal 135° similarity degree C135[i,j] are 

15 calculated, the interpolation processing unit 17 compares 
the similarity along the diagonal 45° direction and the 
similarity along the diagonal 135° direction manifesting 
at the interpolation target pixel "(hereafter referred to 
as the "diagonal similarity") based upon the diagonal 45° 

20 similarity' degree C45[i,j] and the diagonal 135° 

similarity degree C135[i,j] (FIG. 4 S7). Then, it sets 
one of the following values for an index DN[i,j] which 
indicates the diagonal similarity based upon the results 
of the comparison. 

25 [0085] 
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For instance, if; |C45[i,j] - C135[i,j]| > T2 and 
C45[i,j] < C135[i,j] are true with regard to a given 
threshold value T2, the interpolation processing unit 17 
judges that a more marked similarity is manifested along 
5 the diagonal 45° direction than along the diagonal 135° 
direction and sets 1 for the index DN[i,j] (FIG.. 4 S8), 
df; |C45[i,j] - C135[i,j]| > T2 and C45[i,j] > C135[i,j] 
are true, the interpolation processing unit 17 judges that 
a more marked similarity is manifested along the diagonal 

10 135° direction than along the diagonal 45° direction and 
sets -1 for the index DN[i,j] (FIG. 4 S9) and if; 
|C45[i,j] - C135[i,j]| ^ T2 is true, the interpolation 
processing unit 17 judges that the 'degree of similarity 
manifesting along the diagonal 45° direction and along the 

15 diagonal 135° direction are essentially the same and sets 
0 for the index DN[i,j] (FIG. 4 SIO) . 
[0087] 

It is to be noted that the threshold value T2 is used 
to prevent an erroneous judgment that the similarity along 
20 either direction is more marked from being made due to 
noise. 

Next, the interpolation processing unit 17 ascertains 
the specific values of the index HV[i,j] indicating the 
vertical/horizontal similarity and the index DN[i,j] 
25 indicating the diagonal similarity (FIG. 4 Sll) and 
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classifies the class of the similarity manifesting at the 
interpolation target pixel as one of the following; case 1 

case 9. 
[0088] 

5 case 1: (HV[i,j],, DN[i,j]) = (1, 1): marked 

similarity manifesting along the vertical direction and 
the diagonal 45° direction 

case 2: (HV[i,j], DN[i,j]) = (1, 0): marked 
similarity manifesting along the vertical direction 
10 case 3: (HV[i,j], DN[i,j]) = (1, -1): marked 

similarity manifesting along the vertical^ direction and 
the diagonal 135° direction 

case 4: (HV[i., j], DN[i,j]) = (0, 1): marked 
similarity manifesting along the diagonal 45° direction 
15 case 5: (HV[i,j], DN[i,j]) = (0, 0): marked 

similarity manifesting along all the directions or little 

similarity manifesting along all the directions 

[0089] 

case 6: (HV [i, j ] , DN [i, j ] ) = (0, -1): marked 
20 similarity manifesting along the diagonal 135° direction 
case 7: (HV[i,j], DN[i,j]) - {-1, 1): marked 
similarity manifesting along the horizontal direction and 
the diagonal 45° direction 

case 8: (HV[i,j], DN[i,j]) = (-1, 0): marked 
25 similarity manifesting along the horizontal direction 



100 



Translation of 
JPll-363007 



case 9: (HV[i,j], DN[i,j]) = (-1, -1): . marked 
similarity manifesting along the horizontal direction and 
the diagonal 135° direction. 

FIG. 7 illustrates the directions along which marked 
5 similarity manifests^ as indicated by the values of 
HV[i,j], DN[i,j] 
[0090] 

■In FIG. 7, there is no directional indication that 
corresponds to "case 5: (HV[i,j], DN[i,j]) , = (0, 0)." A 

10 marked similarity manifesting along all the directions or 
only a slight similarity manifesting along all the 
directions as in case 5 means that the interpolation 
target pixel is contained within a flat area or is an 
isolated point (an image area manifesting a lower degree 

15 of similarity to nearby pixels and having a high spatial 
frequency) . 

Next, the interpolation processing unit 17 calculates 
the green color interpolation value G[i,j]' as indicated 
below based upon the results of the judgment explained 
2 0 above. 
[0091] 

In case 1, G[i,j] = Gv45[i,j] : FIG. 4 S12 
In case 2, G[i,j] = Gv[i,j] : FIG. 4 S13 
In case 3, G[i,j] = Gvl35[i,j] : FIG. 4 S14 
25 In case 4, G[i,j] = (Gv45[i,j] + Gh45[i,j] ) /2 : FIG. 
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4 S15 



In case 5, G[i,j] = (Gv[i,j] + Gh[i,j] ) /2 



FIG. 



S16 



In case 6, G[i,j] = (Gvl35[i,j] + Ghl35[i,j] ) /2 : 
5 FIG. 4 S17 

In case 7, G[i,j] = Gh45[i,j] : FIG. 4 S18 
In case 8, G[i,j] = Gh[i,j] : FIG. 4 S19 
In case 9, G[i,j] = Ghl35[i,j] : FIG. 4 S20, with 
Gv[i, j]={G[i, j-l]+G[i, j+1] ) /2 
10 +(2-Zti, j]-Z[i, j-2]-Z[i, j+2] ) /8 

+(2-G[i-l, j]-G[i-l, j-2]-G[i-l, j+2] 

+2-G[i+l, j]-G[i+l, j-2]-G[i+l, j+2] ) /16 ... formula 43 
Gv45[i, j]=(G[i, j-l]+G[i, j+1] )/2 

+(2-Z[i, j]-Z[i, j-2]-Z[i, j+2] ) /8 
15 + {2-Z [i-1, j+1] -Z [i-1, j-1] -Z [i-1, j+3] 

+2-Z [i+1, j-1] -Z [i + 1, j-3] -Z [i+1, j+1] ) /16 ... formula 44 
Gvl35[i, j]=(G[i, j-l]+G[i, j+1] ) /2 

+(2-Z[i,j]-Z[i,j-2]-Z[i,j+2])/8 • 
+(2-Z [i-1, j-l]-Z[i-l, j-3]-Z[i-l, j+1] 
20 +2'Z[i + l, j+l]-Z[i + l, j-l]-Z[i+l,j+3] )/16 ... formula 45 

Gh[i, j]=(G[i-l, j]+G[i+l, j] )/2 

+(2-Z[i, j]-Z[i-2, j]-Z[i+2, j] ) /8 
+(2-G[i, j-l]-G[i-2, j-l]-G[i+2, j-1] 

+2-G[i, j+l]-G[i-2, j+l]-G[i+2, j+1] )/16 ... formula 46 
25 Gh45[i, j]=(G[i-l, j]+G[i+l, j] ) /2 
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+(2-Z [i, j]-Z [i-2, j]-Z [i+2, j] ) /8 
+(2-Z[i+l, j-l]-Z [i-1, j-1] -Z [i+3, 

+2-Z j+1] -Z [i-3, j+1] -Z [i + 1, j+1] ) /16 ... formula 47 

Ghl35[i, j]=(G[i-l, j]+G[i+l, j] ) /2 
5 +(2-Z.[i, j]-Z[i-2, j] -Z [i+2, j] ) /8 

+ (2-Z[i-l, j-1] -Z [i-3, j-1] -Z [i + 1, j-1] 

+ 2'Z [i + 1, j+1] -Z [i-1, j+1] -Z [i + 3, j + 1] ) /16,.. formula 48 
[0092] 

FIG. 8 shows the positions of the color information 
10 used to calculate the green color interpolation value 

G[i,j]. In FIG. 8, the color information at the circled 
pixels is used as a contributing factor in the curvature 
information that constitutes the green color interpolation 
value G [i, j ] . 

15 It is to be noted that Gv[i,j]and Gh[i,j] may be 

calculated with following formulas to simplify the 
calculation prodcessing . 
[0093]* 

Gv[i, j] = (G[i, j-l]+G[i, j+1] ) /2 
20 + (2-Z [i, j ] -Z [i, j-2] -Z [i, j+2] ) /4 formula 44' 

Gh[i,.j] = (G[i-l, j]+G[i+l, j] ) /2 

+ (2-Z [i, j ] -Z [i-2, j ] -Z [i+2, j ] ) /4 formula 46' 

In each of formulae 43 - 48, the first term 
constitutes the "average information of the green color 
25 component" and corresponds to the primary terms in 
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formulae 1 and 2. The second term represents the 
"curvature information based upon a color component 
matching a color component at the interpolation target 
pixel" and the third term represents the "curvature 
5 information based upon a color component other than the 
color component at the interpolation target pixel." 
[0094] 

In other words^ in the first embodiment, the average 
information of the green color component is corrected by 

10 using the "curvature information based upon a color 
component matching the color component at the 
interpolation target pixel" and the "curvature information 
based upon a color component other than the color 
component at the interpolation target pixel." 

15 For instance, when marked similarity manifests along 

the diagonal directions and the green color interpolation 
value is calculated by using Gv45[i,j], Gvl35[i,j], 
Gh45[i,j] and Ghl35[i,j] (case 1, case 3, case 4, 'case 6, 
case 7 or case 9) , the average information of the green 

20 color component (the primary term) is corrected by using 
the local information based upon the red color component 
and the curvature information based upon the blue color 
component at phases that are opposite from each other. 
[0095] 

25 Thus, even when the color information corresponding 



104 



Translation of 
JPll'363007 



to the red color component and the color information 
corresponding to the blue color component are offset 
relative to the color information corresponding to the 
green color component due to magnification chromatic 
5 aberration, as illustrated in FIG. 9(1) (equivalent to a 
drawing achieved by superimposing FIG. 16(2) on FIG. 
16(3)), the primary term is corrected in correspondence to 
the average quantity of change in the color information 
corresponding to the red color component and the color 

10 information corresponding to the blue color component, as 
in FIG. 9(2). As a result, by adopting the first 
embodiment, the primary term can be corrected for a 
desired pixel even if there is magnification chromatic 
aberration at the photographic optical system 12, with the 

15 overshoot and the undershoot occurring as a result of the 
G interpolation processing disclosed in USP No. 5,629,734 
canceled out by each other. Consequently, the occurrence 
of color artifacts attributable to over correction can be 
reduced in the first embodiment. 

20 [0096] 

It is to be noted that while an overshoot may also 
occur when correcting a primary term constituted of color 
information corresponding to the blue color component as 
well as when correcting a primary term constituted of 
25 color information corresponding to the red color component. 
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overshoot values corresponding to the individual color 
components are averaged in the first embodiment and thus, 
the average value does not exceed an overshoot value 
resulting from the G interpolation processing disclosed in 
5 USP No, 5,629,734. In addition, even if an undershoot 

occurs when correcting a primary term constituted" of color 
information corresponding to the blue color component or 
correcting a primary term constituted of color information 
corresponding to the red color component, the undershoot 
10 value in the first embodiment never exceeds the undershoot 
value resulting from the G interpolation processing 
disclosed in USP No. 5,629,734. 
[0097] 

In the first embodiment, the image data to undergo 
15 the G interpolation processing are arranged in a Bayer 
array as shown in FIG. 2, with the color information 
corresponding to .the red color component and the color 
information corresponding* to the blue color component 
positioned diagonally to each other. Thus, if color 
20 information corresponding to the blue color component is 
provided at the interpolation target pixel, for instance, 
the local curvature information based upon the red color 
component to be used to correct the primary term is 
calculated by using color information corresponding to the 
25 red color component at pixels positioned along a diagonal 
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direction along which marked similarity to the 
interpolation target pixel manifests. In addition, the 
green color interpolation value is calculated by using the 
color information at pixels set along a diagonal direction 
5 distanced from the interpolation target pixel such as Z[i- 
l,j+3] and Z[i+l,j-3] in formula 44, Z[i-l,j-3] and 
Z[i+l,j+3] in formula 45, Z[i+3,j-l] and Z[i-3,j+l] in 
formula 47 and Z[i-3,j-l] and Z[i+3,j+l] in formula 48. 
[0098] 

10 As a result, in the G interpolation processing in the 

first embodiment which requires a highly accurate judgment 
on the diagonal similarity, the interpolation processing 
unit 17 achieves a high degree of accuracy in the 
judgement of the diagonal similarity by using a plurality 

15 of sets of color information when calculating a plurality 
of types of similarity degree components along the 
diagonal 45° direction and the diagonal 135° direction. 

In other words, the accuracy of the interpolation 
processing is improved through a highly accurate judgment 

20 on the diagonal similarity in the first embodiment. 
[0099] 

In addition, if marked similarity manifests along the 
vertical direction or the horizontal direction and thus 
the green interpolation value is calculated using Gv[i,j] 
25 or Gh[i,j] (case 2 or case 8), curvature information based 
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upon the green color component is used as the "curvature 
information based upon a color component other than the 
color component at the interpolation target pixel" and the 
average; information of the green color component is 
corrected by using curvature information based upon the 
red color component or the blue color component and 
curvature information based upon the green color component 
[0100] 

Under normal circumstances, the effect of 
magnification chromatic aberration often manifests as a 
large offset between the red color component and the green 
color component and a slight offset between the blue color 
component and the green color component. Thus, if color 
information corresponding to the red color component is 
provided at the interpolation target pixel, the curvature 
information based upon the green color component can be 
used as a component at a phase opposite from the phase of 
the curvature information based upon the red color 
component to reduce the occurrence of color artifacts 
resulting from over correction. In addition, since the 
effect of a magnification chromatic aberration is less 
evident if color information corresponding to the blue 
color component is present at the interpolation target 
pixel, color artifacts occur less readily as a result of 
over correction. 
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[0101] 

In the following explanation of the RB interpolation 
processing operation, the RB interpolation processing 
implemented in the prior art is first described and then 
5 the R interpolation processing in FIG. 5 in the RB 
interpolation processing implemented in the first 
embodiment is explained (an explanation of the B 
interpolation processing is omitted) . 

A known example of the RB interpolation processing in 

10 the prior art is linear interpolation processing 

implemented in a color difference space in which after 
calculating color differences at all the pixels (values 
each obtained by subtracting the value indicated by color 
information corresponding to the green color component 

15 from the value indicated by color- information 

corresponding to the red color component (or the blue 
color component) ) , one of the three different types of 
processing (1) (3) described below is implemented on 
each interpolation target pixel to calculate the 

20 interpolation value. 
' [0102] 

(1) If a color component missing at the interpolation 
target pixel is present at the two pixels adjacent to the 
interpolation target pixel along the vertical direction, 
25 the interpolation target value is calculated as a value 
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achieved by adding the color information corresponding to 
the green color component at the interpolation target 
pixel to the average of the color differences at the two 
pixels . 

(2) If a color component missing at the 'interpolation 
target pixel is present at the two pixels adjacent to the 
interpolation target pixel along the horizontal direction, 
the interpolation target value is calculated as a value 
achieved by adding the value indicated by the color 
information corresponding to the green color component at 
the interpolation target pixel to the average of the color 
differences at the two pixels. 

[0103] 

(3) If a color component missing at the interpolation 
target pixel is present at the four pixels adjacent to the 
interpolation target pixel along the diagonal directions, 
the interpolation target value is calculated as a value 
achieved by adding the value indicated by the color 
information corresponding to the green color component at 
the interpolation target pixel to the average of the color 
differences at the four pixels. 

In addition, interpolation processing that 
incorporates nonlinear median processing, which is more 
effective in preventing color artifacts compared to linear 
processing in a color difference space is also implemented 
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in the prior art. 
[0104] 

In the art disclosed in USP No. 5,799,113, in which 
video signals provided in one of the following 
5 colorimetric systems, RGB, YUV and YCbCr, undergo culled 
compression at a resolution of 1/4 to reduce the 
transmission volume, nonlinear median processing is 
implemented to restore the video signals to the original 
resolution by interpolating 3-component data at the culled 

10 pixels which have been lost. For instance, if the video 

signals are provided in the YCbCr colorimetric system, the 
interpolation values for the luminance component Y and the 
interpolation values corresponding to the color components 
Cb and Cr at the culled pixels marked O, A and X in FIGS. 

15 10(1) - IOC are calculated through identical arithmetic 

processing. It is to be noted that in order to retain the 
structure at an edge, the pixel, marked X alone is 
interpolated by using the "median value (median) of the 
values at four nearby pixels, the pixels marked O are each 

20 interpolated by using the average of the values at the 
pixels adjacent along the horizontal direction and the 
pixels marked A are each interpolated by using the average 
of the values at the pixels adjacent along the vertical 
direction. 

25 [0105] 
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However, while the interpolation processing 
implemented as described above is effective in restoring 
the image quality in a dynamic image, it is not suited for 
processing a still image that requires high definition. 
5 Namely, the art disclosed in USP No. 5,799,113, in which 
the luminance component- Y and the color components Cr and 
Cb are handled in exactly the same way, achieves only a 
very low degree of accuracy with regard to the 
interpolation values for the luminance component Y which 

10 determines the resolution. In addition, since the 

luminance component Y is interpolated by using the median, 
the likelihood of the image structure becoming lost is 
high. Furthermore, there is a concern that color 
artifacts may spread when the data are converted to the 

15 RGB colorimetric system. 
[0105] 

In an electronic camera which employs an image- 
capturing sensor constituted by arranging R, G and B color 
filters in a Bayer array to generate a still image, the 

20 interpolation processing on the green color component 

which is equivalent to the luminance component with a high 
spatial frequency (G interpolation processing) can be 
implemented with a very high degree of accuracy by using 
similarity manifesting between the interpolation target 

25 pixel and nearby pixels and calculating the interpolation 
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value using a plurality of color components, as explained 
earlier. In such an electronic camera, after implementing 
high-definition interpolation processing on the green 
color component, which most faithfully reflects the high- 
frequency information in the image data, the interpolation 
processing on the red color component and the blue color 
component is achieved through linear interpolation in 
color difference spaces relative to the green color 
component to reduce color artifacts by reflecting the 
high-frequency information in the image data in the red 
color component and the blue color component. 
[0107] 

For instance, if the sets of color information at 
individual pixels are arranged one-dimensionally in the 
order of (Rl, G2, R3) , the red color interpolation value 
is calculated through; 

R2= (R1+R3) /2+ (2 • G2-G1-G3) /2 ... formula 49. In the formula, 
G2 represents color information corresponding to the green 
color component in the original image and Gl and G3 each 
represent a green color interpolation value obtained 
through the G interpolation processing. 
[0108] 

However, this RB interpolation processing poses a 
problem in that the color artifact is allowed to remain in 
the vicinity of an isolated point (an image area 
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manifesting only slight similarity to nearby pixels and 
having a high spatial frequency) . In the prior art, this 
type of color artifact is often eliminated in post 
processing, in which a and b hue planes obtained by 
converting the image data to the Lab colorimetric system 
individually undergo median filtering after the G 
interpolation processing and the RB interpolation 
processing are implemented. 
[0109] 

Since a 3 x 3 (=9 points) filter size achieves 
hardly any effect, the filter size must be set over a 
large range of 5 x 5 (=25 points) when applying such a 
median filter. 

In other words, in the electronic camera described 
above, extremely heavy processing must be implemented 
since both the RB interpolation processing in the prior 
art and the median processing must be performed in the 
interpolation processing on the red color component and 
the blue color component in a still image and also the 
filter size must be set over a wide range for the median 
processing. Furthermore, the risk of the fine structure 
in a colored area (hereafter referred to as a "color 
structure") being lost is higher when the filter size in 
the median processing is increased. 
[0110] 
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Accordingly, in the first embodiment, RB 
interpolation processing through which red color and blue 
color interpolation values can be calculated quickly with 
a high degree of accuracy without allowing any color 
5 artifacts to remain in the vicinity of an isolated point 
or losing the color structure is proposed. It is to be 
noted that the following is an explanation of the only R 
interpolation processing in the RB interpolation 
processing, given in reference to FIG. 5. 

10 First, the interpolation processing unit 17 

calculates a color difference that contains the red color 
component for each pixel at which color information 
corresponding to the red color component is present by 
subtracting the green color interpolation value (the 

15 value obtained through the G interpolation processing 

explained earlier) from the value indicated by the color 
information corresponding ^ to the red color component. (FIG. 
5 SI) . ~ • ^ 

[0111] 

20 For instance, the interpolation processing unit 17 

calculates a color difference Cr[i,j] containing the red 
color component at a pixel at given coordinates [i,j] with 
color information corresponding to the red color component 
as; Cr [i, j ] =R [i, j ] -G [i, j ] ... formula 50. 

25 It is to be noted that in the first embodiment, when 
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the color differences containing the red color component 
have been calculated as described above, the color 
differences containing the red color component are set so 
as to surr,ound pixels at which color information 
5 corresponding to the red color component is missing and 
color information corresponding to the blue color 
component is present from the four diagonal directions. 
[0112] 

The interpolation processing unit 17 interpolates the 
10 color difference containing the red color component for 
each of the pixels surrounded by color differences 
containing the red color component from the four diagonal 
directions (each pixel at which color information 
corresponding to the red color component is missing and 
15 color information corresponding to the blue color 

component is present in the first embodiment) by using the 
median of the color differences containing the red color 
component at the pixels set diagonally to the target pixel 
(FIG. 5 S2) . . 

20 Namely, in the first embodiment, the interpolation 

processing unit 17 calculates the color difference Cr[m,n] 
at the pixel at given coordinates [m,n] surrounded by 
color differences containing the red color component from 
the four diagonal directions as shown in FIG. 11(1) 

25 through; 
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Cr [m, n] =inedian{Cr [m-1, n-1] , Cr [m+1 , n-1 ] , 

Cr [m-1, n+1] , Cr [m+l, n+1] } ... formula 51. 
-In the formula, median { } represents a function through 
which the median of a plurality of elements is calculated 
5 and, if there are an even number of elements, it takes the 
average of the two middle elements. 
[0113] 

In the first embodiment, when the color differences 
containing the red color component have been calculated 

10 through formulae 50 and 51, the color differences 

containing the red color component are set so as to 
surround pixels at which color information corresponding 
to the red color component and color information 
corresponding to the blue color component are both missing 

15 from the four directions; i.e., from above, from below and 
from the left and the right. 

The interpolation processing unit 17 interpolates the 
color difference containing the^ red color component for 
each of the pixels surrounded by color differences 

20 containing the red color component from the four 

directions; i.e.,. from above, from below and from the left 
and the right (each pixel at which color information 
corresponding to the red color component and color 
information corresponding to the blue color component are 

25 ' both missing in the first embodiment) by using the median 
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of the color differences containing the red color 
component at the pixels set above, below and to the left 
and the right of the pixel (FIG. 5 S3) . 
[0114] 

Namely, in the first embodiment, the interpolation 
processing unit 17 calculates the color difference Cr[m,n] 
at the pixel at given coordinates [m,n] surrounded by 
color differences containing the red color component from 
the four directions; i.e., from above, from below and from 
the left and the right as shown in FIG. IIB through; 
Cr [m, n] =median{Cr [m, n-1] , Cr [m-1, n] , 

Or [m+l, n] , Cr [m, n+1] } ... formula 52 

[0115] 

Next, the interpolation processing unit 17 converts 
the color difference containing the red color component 
calculated through formula 51 or formula 52 for each pixel 
at which color information corresponding to the red color 
component is missing to a red color interpolation value by 
using color information corresponding to the green color 
component (or the green color interpolation value) (FIG. 5 
S4) . 

Namely, the interpolation processing unit 17 
calculates the red color interpolation value R[m,n] for 
the pixel at given coordinates [m^n] through; 
R [m, n] =Cr [m, n] +G [m, n] ... formula 53. 
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[0116] 

■The median processing described above is implemented 
on the color differences representing the hue alone and is 
not implemented on the luminance component. In addition, 
when the pixel marked O in FIG. 12(1) is the interpolation 
target pixel in the R interpolation processing, the color 
differences containing the red color component at the 
pixels marked X are calculated by using the color 
differences Cr over a 3 x 5 range, and thus, the color 
difference containing the red color component at the pixel 
marked O represents a value which is close to the results 
of median processing implemented by weighting the color 
differences Cr within the 3x5 range. When the pixel 
marked A in FIG. 12(2) is the interpolation target pixel, 
on the other hand, the color differences containing the 
red color component at the pixels marked X are calculated 
by using the color differences Cr over a 5 x 3 range, and 
thus, the color- difference containing the red color ' 
component at the pixel marked A represents a value which 
is close to the results of median processing implemented 
by weighting the color differences Cr within the 5x3 
range . 
[0117] 

In other words, in the first embodiment, advantages 
substantially similar to those achieved through median 
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processing implemented over a wide range are achieved 
while keeping down the filter size. As a result, by 
adopting the first embodiment, the occurrence of color 
artifacts around an isolated point is reduced without 
5 destroying the color structure. Thus, a great improvement 
is achieved in the color artifact reduction effect over 
the art disclosed in USP No. 5,799,113. 
[0118] 

In addition, since the color differences at only four 

10 points are each used in the median, processing in FIG. 5 S2 
and FIG. 5 S3 in the first embodiment, good processing 
efficiency is achieved and extremely fast median 
processing is enabled. 

It is to be noted that while the RB interpolation 

15 processing is implemented after the G interpolation 
processing in the first embodiment, RB interpolation 
processing similar to that in the embodiment can be 
implemented without having to perform G interpolation 
processing on image data provided in the YCbCr 

20 colorimetric system with Y, Cb and Cr culled at a ratio of 
4:1:1 since the luminance component Y is left intact in 
the image data. 
[0119] 

(Second Embodiment) 

25 The following is an explanation of the operation 
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achieved in the second embodiment. 

It is to be noted that since the RB interpolation 
processing in the second embodiment is implemented as in 
the first embodiment, its explanation is omitted. However, 
5 in the second embodiment, color differences containing the 
red color component are interpolated for some of the 
pixels at which color information corresponding to the 
green color component is present (equivalent to the pixels 
provided with G color filters) through formula 51 

10 presented earlier, and color differences containing the 
red color component are interpolated for the remaining 
pixels at which color information corresponding to the 
green color component is present and for the pixels at 
which color information corresponding to the blue color 

15 component is present through formula 52. 
[0120] 

The following is an explanation of the G 
interpolation processing. 

Since the closest pixels at which the green color 
20 component (the closest green color component) is present 

are located along the horizontal direction relative to the 
pixel to undergo the G interpolation processing as shown 
in FIG. 3 in the second embodiment, it is not necessary to 
calculate the similarity degrees or to judge the direction 
25 along which similarity manifests as required in the first 



121 



Translation of 
JPll-363007 



embodiment during the G interpolation processing. However, 
the calculation of similarity degrees and judgment with 
regard to the direction along which similarity manifests 
may be performed along the diagonal 45° direction and the 
5 diagonal 135° direction in which the second closest pixels 
with green color component (the second closest green color 
component) are present. 
[0121] 

In the second embodiment, the interpolation 
10 processing unit 17 calculates the green color 

interpolation value G[i,j] through the following formula 
54 based upon the image data arranged as shown in FIG. 3. 
G[i, j]=(G[i-l, j]+G[i+l, j] ) /2 

+ (2*Z[i, j]-Z [i-2, j]-Z [i+2, j],) /8 
15 +(2-Z[i, j-l]-Z[i-2, j-l]-Z[i+2, j-1] 

+ 2-Z [i, j+1] -Z [i-2, j + 1] -Z [i + 2, j + 1] ) /16 ... formula 54 
In formula 54, the first term represents the "average 
information of the green color component", which is 
equivalent to the primary terms in formula 1 and formula 2. 
20 In addition, while the second term represents the 

"curvature information based upon a color component 
matching the color component at the interpolation target 
pixel" and the third term represents the "curvature 
information based upon a color component other than the 
25 color component at the interpolation target pixel", the 
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third term constitutes the "curvature information based 
upon the blue color component" if color information 
corresponding to the red color component is present at the 
interpolation target pixel (FIG. 3(1)), whereas the third 
5 term constitutes the "curvature information based upon the 
red color component" if color information corresponding to 
the blue color component is present at the interpolation 
target pixel (FIG. 3(2) ) . 
[0122] 

10 In other words, the average information of the green 

color component- (the primary term) is corrected by using 
the curvature information based upon the red color 
component and the curvature information based upon the 
blue color component at phases opposite from each other in 

15 the second embodiment. 

As a result, as in the first embodiment, even when 
the color information corresponding to the red color 
component and the color information corresponding to the 
blue color component are offset relative to the color 

20 information corresponding to the green color component due 
-to a magnification chromatic aberration, the primary term 
is corrected in correspondence to the average change in 
quantity in the color information corresponding to the red 
color component and the color information corresponding to 

25 the blue color component in the second embodiment (see FIG. 
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9) . Thus/ even when there is a magnification chromatic 
aberration at the photographic optical system 12, the 
primary term can be corrected for a desired pixel with the 
overshoot and undershoot occurring in the G interpolation 
5 processing disclosed in USP No. 5,629,734 canceling out 
each other in the second embodiment. Consequently, the 
occurrence of color artifacts due to over correction can 
be reduced by adopting the second embodiment. 
[0123] 

10 (Third Embodiment) 

The following is an explanation of the operation 
achieved in the third embodiment. 

It is to be noted that since the RB interpolation 
processing in the third embodiment is implemented as in 

15 the first embodiment, its explanation is omitted. 

In the following explanation of the G interpolation 
processing, operating details identical to those in the 
first embodiment are not explained.' It is to be noted 
that the difference between the G interpolation processing 

20 in the third embodiment and the G interpolation processing 
in the first embodiment is in the operation performed 
after a judgment is made with regard to degrees of 
similarity manifested by the interpolation target pixel. 
For this reason, a flowchart of the operation performed at 

25 the interpolation processing unit 17 during the G 
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interpolation processing in the third embodiment is not 
provided. 
[0124] 

The interpolation processing unit 17 ascertains 
5 degrees of similarity manifesting by the interpolation 
target pixel as in the first embodiment (corresponds to 
FIG. 4 SI Sll) and classifies the type of similarity at 
the interpolation target pixel as one of case 1-9 
explained earlier . 
10 Then, the interpolation processing unit 17 calculates 

the inclination Gk[i,j] of the green color component and 
the inclination Zk:[i,j] of the red color component (or the 
blue color component) relative to the direction 
perpendicular to the direction judged to manifest marked 
'15 similarity as indicated below. 
[0125] 

In easel, 

Gk[i, j]=( (G[i-1, j]+G[i, j-1] )-(Gti, j+l]+G[i+l, j] ) )72 
••• formula 55 

20 Zk[i, j]-( (Z[i-2, j]+Z[i, j-2] )-(Z[i, j+2]+Z[i+2, j] ) ) /2 

formula 56 
In case2, 

Gk[i, j]=G[i, j-l]-G[i, j+1] — formula 57 
Zk[i, j]=Z[i, j-2]-Z[i, j+2] formula 58 
25 In caseS, 
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Gk[i, j] = { (G[i-1, j]-t-G[i, j+1] )-(G[i, j-l]+G[i + l, j] ) )/2 
••• formula 59 

Zk[i, j] = ( (Z[i-2, j]+Z[i, j+2] )-(Z[i, j-2]+Z[i + 2, j] ) ) /2 

;•* formula 60 
5 In case A, same as an case 1 

In case 5, 
Gk[i,j]-1, Zk[i,j]=l 

In case 5, same as an case 3 

In case 7, same as an case 1 
1,0 In case 8 

Gk[i, j]=G[i-l, j]-G[i + l, j] formula 61 
Zk [i, j ] =Z [i-2, j ] -Z [i+2, j ] •*• formula 62 

In case 9, same as an case 3 

Next, the interpolation processing unit 17 calculates 
15 the green color interpolation value G[i,j] as follows. 
[0126] 
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Gvk[i, j] = (G[i, j-l]+G[i, j+1] ) /2 

+Gk[i, j]/Zk[i, j] • (2-Z[i, j]-Z[i, j-2]-Z[i, j+2] )/4 
formula 63 and 
Ghk[i, j]=(G[i-l, j]+G[i+l, j] ) /2 
5 +Gk[i, j] /Zk[i, j] • (2-Z[i, j]-Z [i-2, j]-Z[i+2, j] ) /4 

••• formula 64 . 
[0127] 

In formulae 63 and 64, the first term represents the 
"average information of the green color component" which 

10 is equivalent to the primary term in formula 1 and formula 
2. The second term is the "curvature information based 
upon a color component matching the color component at the 
interpolation target pixel" multiplied by a weighting 
coefficient (a value indicating the correlation between 

15 the inclination Gk[i,j] of the green color component and 

the inclination Zk[i,j] of the red color component (or the 
blue color component): Gk [i, j ] /Zk [i, j ] ) , and is equivalent 
to the correctional term.' 
[0128] 

20 Namely, in the third embodiment, the average 

information of the green color component is corrected by 
using the "curvature information based upon a color 
component matching the color component at the 
interpolation target pixel" multiplied by the weighting 

25 coefficient. 
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Now, a problem that arises when calculating the 
correction value by simply adding the "curvature 
information based upon a color component matching the 
color component at the interpolation target pixel" without 
5 being multiplied by the weighting coefficient to the 
"average information of the green color component" is 
discussed . 
[0129] 

For instance, when color information corresponding to 

10 the green color component and color information 

corresponding to the red color component (or the blue 
color component) are provided as indicated by # in FIG. 13 
(when the values indicated by the color information 
corresponding to the green color component start to 

15 increase at a specific position and the values indicated 
by the color information corresponding to the red color 
component (or the blue color component) start to decrease 
at the same position) , the "curvature information based 
upon a color component matching the color component at the 

20 interpolation target pixel" indicates a positive value. 

Thus, in such a case, if the "curvature information based 
upon a color component matching and the color component at 
the interpolation target pixel" is added to the "average 
information of the green color component" without first 

25 multiplying it with the weighting coefficient, the 
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"average information of the green color component", which 
should be corrected along the negative direction, becomes 
corrected in the positive direction as indicated by A in 
FIG. 13, resulting in an overshoot. 
5 [0130] 

In other words, when color information corresponding 
to the green color component and the color information 
corresponding to the red color component (or the blue 
color component) change in opposite directions from a 

10 specific position at a color boundary, an overshoot or an 
undershoot occurs if the correctional term is calculated 
simply by adding the "curvature information based upon a 
color component matching the color component at the 
interpolation target pixel" to the "average information of 

15 the green color component" without first multiplying the 
"curvature information based upon a color component 
matching the color component at the interpolation target 
pixel" by the weighting coefficient: 
[0131] 

20 In the embodiment, if color information corresponding 

to the green color component and color information 
corresponding to the red color component (or the blue 
color component) are provided as indicated by # in FIG. 13, 
-the sign of the inclination Gk[i,j] corresponding to the 

25 green color component and the sign of the inclination 



129 



Translation of 
JP11~363007 



Zk[i,j] corresponding to the red color component (or the 
blue color component) are opposite from each other 
resulting in the weighting coefficient being a negative 
value. Thus, the "average information of the green color 
component" is corrected in the desired direction as 
indicated by □ in FIG. 13, preventing an overshoot or an 
undershoot from occurring. 
[0132] 

Consequently, the occurrence of color artifacts 
attributable to over correction at a color boundary is 
reduced in the third embodiment. 

It is to be noted that while no restrictions are 
imposed with regard to the value of the weighting 
coefficient in the third embodiment, restrictions may be 
imposed to set the value of the weighting coefficient 
within a specific range to ensure that the correctional' 
term does not become too large. 
[0133] 

For instance, the^ range for the weighting coefficient 
may be set; | Gk [i, j ] /Zk [i, j ] 1^5. 
(Fourth Embodiment) 

The following is an explanation of the operation 
achieved in the fourth embodiment. 
[0134] 

It is to be noted that since the RB interpolation 
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processing in the fourth embodiment is performed as in the 
first embodiment, its explanation is omitted. However, in 
the fourth embodiment, color differences containing the 
red color component are interpolated for some of the 
5 pixels at which color information corresponding to the 
green color component is present through formula 51 and 
color differences containing the red color component are 
interpolated for the remaining pixels at which color 
information corresponding to the green color component is 
10 present and for pixels at which color information 

corresponding to the blue color component is present 
through formula 52, as in the second embodiment. 
[0135] 

Now, the G interpolation processing is explained. 

15 In the fourth embodiment, in which the closest pixels 

at which color information corresponding to the green 
color component is present are set along the horizontal 
direction to a pixel to undergo the G interpolation 
processing as shown in FIG. 3, interpolation processing is 

20 achieved in the simplest manner by using the color 
information at the pixels set along the horizontal 
direction. Accordingly, the green color interpolation 
value G[i,j] is calculated in the fourth embodiment as in 
case 8 in the third embodiment. 

25 [0136] 
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Namely, the interpolation processing unit 17 
calculates the green color interpolation value G[i,j] 
through formula 65. 
G[i, j]=(G[i-l, j]+G[i+l, j] ) /2 
5 +Gk[i, j]/Zk[i, j] • (2-Z[i, j]-Z[i-2, j]-Z[i+2, j] )/4 

formula 65, with 
Gk[i, j]=G[i-l, j]-G[i + l, j] formula 61 and 
Zk[i, j ]-Z [i-2, j ] -Z [i + 2, j ] ••• formula 62. 
[0137] 

10 As indicated above, in the fourth embodiment, the 

average information of the green color component is 
corrected by using the "curvature information based upon a 
color component matching the color component at the 
interpolation target pixel" multiplied by the weighting 

15 coefficient (a value representing the correlation between 
the inclination Gk[i,j] corresponding to the green color 
component and the inclination Zk[i,j] corresponding to the 
red color component (or the blue color component) : 
Gk [i, j ] /Zk [i, j ] ) as in the third embodiment. As a result, 

20 the occurrence of color artifacts attributable to over 

correction at a color boundary can be reduced through the 

fourth embodiment. 

[0138] 

(Fifth Embodiment) 

25 The following is an explanation of the operation 
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achieved in the fifth embodiment. 

FIG. 14 is a functional block diagram representing 
the fifth embodiment. 

It is to be noted that the fifth embodiment 
5 corresponds to interpolation processing executed by a 
personal computer by using the recording medium having 
recorded therein an interpolation processing program 
according to one of claims 21-25. 
[0139] 

10 In FIG. 14, the same reference numbers are assigned 

to components achieving identical functions to those in 
the functional block diagram in FIG. 1 to preclude the 
necessity for repeated explanation of their structures. 

The structure of an electronic camera 20 shown in FIG. 

15 14 differs from that of the electronic camera 10 in FIG. 1 
in that a control unit 21 and an image processing unit 22 
in FIG. 14 replace the control unit 11 and the image 
processing unit 15 in FIG. 1, with an interface unit 23 in 
FIG. 14 provided as an additional component. 

20 [0140] 

In addition, in FIG. 14, a personal computer 30 is 
provided with a CPU 31, an interface unit 32, a hard disk 
33 and a memory 34 with the CPU 31 connected to the 
interface unit 32, the hard disk 33, the memory 34, the 
25 CD-ROM drive device 35 and the communication interface 



133 



Translation of 
JPll-363007 



unit 36 via a bus. 

It is to be noted that an interpolation processing 
program (an interpolation processing program for executing 
interpolation processing similar to that implemented at 
5 the interpolation processing unit 17 in the various 

embodiments explained earlier) recorded at a recording 
medium such as a CD-ROM 37 is pre-installed at the 
personal computer 30 via the CD-ROM drive device 35. In 
other words, the interpolation processing program is 
10 stored at the hard disk 33 in an execution-ready state. 
[0141] 

The following is an explanation of the operation 
achieved in the fifth embodiment, given in reference to 
FIG. 14. 

15 First, image data generated as in the electronic 

camera 10 shown in FIG. 1 are provided to the image 
processing unit 22 in the electronic camera 20. The image 
data undergo image processing (e . g .,' gradation conversion 
processing) other than interpolation processing at the 

20 image processing unit 22, and the image data having 

undergone the image processing are then recorded at the 

recording unit 16 in an image file format. 

[0142] 

This image file is provided to the personal computer 
25 30 via the interface unit 23. 
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Upon obtaining the image file via the interface unit 
32, the CPU 31 in the personal computer 30 executes the 
interpolation processing program. 

Namely, interpolation processing similar to that 
5 achieved in the embodiments explained earlier is 

implemented on the personal computer 30 in the fifth 
embodiment . 
[0143] 

[Effect of the Invention] 

10 As described above, the interpolation value is 

calculated by correcting the "average information of the 
first color component" with the "curvature information 
based upon a color component matching the color component 
at the interpolation target pixel" and the "curvature 

15 information based upon a color component other than the 

color component at the interpolation target pixel" in the 
inventions according to claims 1, 4 and 21. As a result, 
the occurrence of color artifacts attributable to over 
correction can be reduced. 

20 [0144] ■ 

In the inventions according to claims 2,. 3,' 5-8 and 
22, the interpolation value is calculated by using color 
information at pixels located along a plurality of 
directions in correspondence to the degrees of similarity 
25 to the interpolation target pixel. In particular, the 
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similarity manifesting along the diagonal directions is 
reflected in the interpolation value calculated in the 
inventions according to claims 5-8 . 

As a result, by adopting any of the inventions 
5 according to claims 2, 3, 5-8, the interpolation value can 
be calculated without loss of information related to the 
structure of the image. 
[0145] 

The adverse effect of the noise can be reduced in the 
10 judgment of the similarity in the invention according to 

claim 9, color information corresponding to a plurality of 
color components is reflected in the judgment of 
similarity in the invention according to claims 10 and 11 
and the continuity with nearby pixels is reflected in the 
15 judgment of similarity in the invention according to claim 
12. . . 

As a result, by adopting any of the invention 
according to claims 9-12, the 'accuracy of the similarity 
judgment is improved. 
20 [0146] 

The direction along which correction is implemented 
and the width of the correction can be adjusted through 
the use of a weighting coefficient by correcting the 
"average information of the first color component" with 
25 the "curvature information based upon a color component 
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matching the color component at the interpolation target 
pixel" multiplied by the weighting coefficient in the 
inventions according to claims 13, 14 and 23. In 
particular, in the invention according to claim 14, in 
5 which the weighting coefficient is calculated in 

correspondence to the correlation among the inclinations 
of color information corresponding to the different color 
components in the area nearby the interpolation target 
pixel, the direction along which the correction is 
10 implemented and the width of the correction can be 

adjusted based upon the correlation when correcting the 
"average information of the first color component." 
[0147] 

Consequently, the occurrence of color artifacts 
15 attributable to over correction at a color boundary can be 
reduced by adopting the invention according to claim 13 or 
14 . 

In any of the inventions according to claims 15-20, 
24 and 25, the hue value of the interpolation target pixel 
20 is calculated by using the median of the hue values at a 
plurality of pixels located near the interpolation target 
pixel. 
[0148] 

Thus, the occurrence of color artifacts near an 
25 isolated point can be reduced without losing the color 
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structure by adopting any of the inventions according to 
claims 15-20. In addition, the interpolation value can be 
calculated quickly compared to the interpolation 
processing in the prior art (the processing achieved by 
5 combining RB interpolation processing and median 
processing) . 

[Brief Description of the Drawings] 

[FIG. 1] FIG. 1 is a functional block diagram of an 
electronic camera corresponding to first through fourth 
10 embodiments; 

[FIG. 2] FIG. 2 shows the arrangements of the color 
components in the image data adopted in the first 
embodiment and the third embodiment; 

[FIG. 3] FIG. 3 shows the arrangements of the color 
15 components in the image data adopted in the second 
embodiment and the fourth embodiment; 

[FIG. 4] FIG. 4 is a flowchart (1) of the operation 
achieved at the interpolation processing' unit in the first 
embodiment; 

20 [FIG. 5] FIG. 5 is a flowchart (2) of the operation 

achieved at the interpolation processing unit in the first 
embodiment; 

[FIG. 6] FIG. 6 illustrates methods of weighted addition 
of similarity degree components; 
25 [FIG. 7] FIG. 7 shows the directions along which marked 
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similarity manifests in correspondence to values (HV[i,j], 
DN[i, j] ) ; 

[FIG. 8] FIG. 8 shows the positions of the color 
information used to calculate the green color 
.5 interpolation value G[i,j]; 

[FIG. 9] FIG. 9 shows how the adverse effect of 
magnification chromatic aberration . is eliminated; 

[FIG. 10] FIG. 10 illustrates median processing of the 
prior art; 

10 [FIG. 11] FIG. 11 illustrates the 'median processing 
operation achieved in the first embodiment; 
[FIG. 12] FIG. 12 illustrates the ranges of the median 
processing implemented in the first embodiment; 
[FIG. 13] FIG. 13 illustrates the function of the 
15 weighting coefficient in the third embodiment; 

[FIG. 14] FIG. 14 is a functional block diagram of a fifth 
embodiment; 

[FIG. 15] FIG. ^15 illustrates an example of the 
interpolation processing in the prior art; 
,20 [FIG. 16] FIGS. 16 illustrates the adverse effects of 
magnification chromatic aberration; 

[FIG. 17] FIGS. 17 illustrates over correction occurring 
due to magnification chromatic aberration; and 
[FIG. 18] FIGS. 18 illustrates the effects of over 
25 correction occurring at a color boundary. 
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[Expla: 


nation of the reference numerals] 


10, 


20 


an electronic camera 


11, 


21 


a control unit 


12 




a photographic optical system 


13 




an image-capturing unit 


14 




an A/D conversion unit 


15, 


22 


an image processing unit 


16 




a recording unit 


17 




an interpolation processing unit 


23, 


32 


an interface unit 


30 




a personal computer 


31 




a CPU '31 


33 




a hard disk 


34 




a memory 
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[TITLE OF DOCUMENT] ABSTRACT OF DISCLOSURE 
[ABSTRACT] 

[PROBLEMS TO BE SOLVED] 

An object of the present invention that relates to an 
5 interpolation processing apparatus that engages in 

interpolation processing on color image data to supplement 
a color component and a luminance component missing in 
pixels and a recording medium having an interpolation 
processing program for achieving the interpolation 

10 processing on a computer, that can be read by a computer, 
is that occurrence of color artifacts can be prevented. 
[MEANS FOR SOLVING THE PROBLEMS] 

An interpolation processing apparatus engages in 
processing on image data which are provided in a 

15 colorimetric system constituted of first - nth (n ^ 3) 
color components and include color information 
corresponding to a single color component provided at each 
pixel to determine an interpolation value equivalent to 
color information corresponding to the first color 

20 component for a pixel at which the first color component 
is missing, and includes: a means for interpolation value 
calculation that uses color information at nearby pixels 
located nearby an interpolation target pixel to undergo 
interpolation processing to calculate an interpolation 

25 value including, at least a first term indicating local. 
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average information with regard to the first color 
component, a second term indicating curvature information 
with regard to a color component matching a color 
component at the interpolation target pixel and a third 
5 term indicating curvature information with regard to a 
color component other than the color component at the 
interpolation target pixel. 
[REFERENCE] FIG. 4 



142 



CLJ> 



CD 



CD 



UL. 
CD 



CD 



C-3 



CD 
CD 



CD 



CZ5 

a. 



-cc ck£ cr> 

^ CD LL- 
O CD 



CD 




< 
a: 

LU 

< 
o 

o 



o 
tr 
I- 
o 

LU 

_J 

LU 



o 

Q 

or 

CC3 



CO 



LU 
O 



CO 
CO 
UJ 

o 



6t 



to 



O 

Q.UJ 



O 
00 

UJ 
<03 




FIG.2 



THE ARRANGEMENTS OF THE COLOR COMPONENTS IN 
THE IMAGE DATA ADOPTED IN THE FIRST EMBODIMENT AND 
THE THIRD EMBODIMENT 
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THE ARRANGEMENTS OF THE COLOR COMPONENTS IN 
FIG 3 """"^ "^^^ ^^'^^ ADOPTED IN THE SECOND EMBODIMENT AND 

THE FOURTH EMBODIMENT 
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A FLOWCHART (1) OF THE OPERATION ACHIEVED 
AT THE INTERPOLATION PROCESSING UNIT 
IN THE FIRST EMBODIMENT 
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A FLOWCHART (2) OF THE OPERATION ACHIEVED 
AT THE INTERPOLATION PROCESSING UNIT 
IN THE FIRST EMBODIMENT 
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INFORMATION USED TO CALCULATE 
THE GREEN COLOR 
INTERPOLATION VALUE G[l, J] 
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case4 : (HV,DN)=(0, 1) caseS : (HV,DN)=(0,O) case6 : (HV,DN)=(0,-1) 
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case? : (Hy,DN)=(-l, 1) caseS : (HV,DN)=(-1,0) case9 : (HV,DN)=(-1,-1) 
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FIG. 11 



ILLUSTRATES THE MEDIAN PROCESSING OPERATION 
ACHIEVED IN THE FIRST EMBODIMENT 
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FIG.12 ILLUSTRATES THE RANGES OF THE MEDIAN PROCESSING 

IMPLEMENTED IN THE FIRST EMBODIMENT 
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FIG. 13 



ILLUSTRATES THE FUNCTION OF THE WEIGHTING 
COEFFICIENT IN THE THIRD EMBODIMENT 
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ILLUSTRATES THE ADVERSE EFFECTS OF MAGNIFICATION 
CHROMATIC ABERRATION 
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ILLUSTRATES OVER CORRECTION OCCURRING DUE TO 
MAGNIFICATION CHROMATIC ABERRATION 
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ILLUSTRATES THE EFFECTS OF OVER CORRECTION 
OCCURRING AT A COLOR BOUNDARY. 
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